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Abstract 

In  recent  years,  the  Federal  Government  has  attempted  to  utilize  strategic  sourcing 
to  reduce  acquisition  and  operating  expenses.  Currently  accepted  best  practices  for 
implementing  strategic  sourcing  of  services  and  commodities  developed  in  the  private 
sector  fail  to  account  for  the  diverse  and  unique  set  of  strategic  objectives  present  in 
public  sector  acquisitions.  Value  Focused  Thinking  (VFT)  was  used  to  develop  a 
hierarchy  of  values  and  objectives  to  assist  the  Air  Force  Civil  Engineer  Commodity 
Council  (CECC)  in  assessing  opportunities  for  the  strategic  sourcing  program.  This 
hierarchy  represents  the  full  range  of  program  objectives,  and  was  used  to  develop  a 
value  function  useful  for  systematically  evaluating  service  and  commodity  requirements 
for  strategic  sourcing  potential.  In  addition,  a  comparative  study  was  conducted  between 
the  results  obtained  with  the  new  VFT  model  and  the  results  of  the  existing  opportunity 
assessment  process. 
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USING  VALUE-FOCUSED  THINKING  AS  AN  ALTERNATIVE  MEANS  OF 


OPPORTUNITY  ASSESSMENT  FOR  STRATEGIC  SOURCING  APPLICATIONS 

I.  Introduction 


As  the  Department  of  Defense  (DOD)  seeks  to  streamline  acquisition  process  in 
an  effort  to  decrease  overall  expenses,  strategic  sourcing  programs  have  been  established 
to  develop  enterprise  solutions  that  will  better  leverage  the  buying  power  of  the 
government  in  order  to  achieve  lower  costs.  In  the  Air  Force,  obstacles  to  conducting  a 
thorough  assessment  of  all  possible  enterprise  contract  opportunities  have  prevented  such 
an  analysis  from  occurring.  This  research  project  seeks  to  overcome  these  obstacles 
through  the  development  of  an  accurate  and  usable  means  of  assessing  strategic  sourcing 
opportunities  in  order  to  further  advance  cost  reduction  goals  in  the  DoD. 

Background 

At  the  present  moment,  the  nation  is  in  the  midst  of  a  fiscal  crisis.  The  Federal 
government’s  expenses  over  the  past  decade  have  far  exceeded  revenues  due  to  a  variety 
of  factors.  Among  these  are  rising  costs  of  entitlement  programs,  a  decade  of  armed 
conflict,  and  government  actions  to  soften  the  blow  of  a  global  recession.  Regardless  of 
the  external  factors  that  have  contributed  to  the  recent  explosion  of  national  debt,  the  fact 
remains  that  the  current  fiscal  course  of  the  United  States  is  unsustainable.  According  to 
the  Congressional  Budget  Office,  if  current  policies  are  allowed  to  proceed  unchanged, 
the  national  debt  will  balloon  to  over  ninety  percent  of  the  nation’s  Gross  Domestic 
Product  in  the  coming  decade,  and  will  expand  to  two  hundred  percent  in  the  year  2037 
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(Congressional  Budget  Office,  2012).  This  crisis  has  elicited  reactions  from  leaders  both 
within  and  outside  of  the  federal  government,  but  perhaps  no  governmental  organization 
has  made  as  much  of  an  effort  to  prepare  for  the  pending  changes  in  public  policy  as  the 
DoD.  In  fact,  the  chairman  of  the  Joint  Chiefs  of  Staff  recently  deemed  the  national  debt 
the  number  one  threat  to  the  national  security  of  the  United  States  (Armed  Forces  Press 
Service,  2011). 

In  2005,  the  Office  of  Management  and  Budget  released  a  memorandum 
requiring  all  federal  agencies  to  take  steps  to  implement  a  strategic  sourcing  program 
with  the  overall  goal  of  reducing  the  cost  of  government  operations.  This  memo  defined 
strategic  sourcing  as  “the  collaborative  and  structured  process  of  critically  analyzing  an 
organization’s  spending  and  using  this  information  to  make  business  decisions  about 
acquiring  commodities  and  services  more  effectively  and  efficiently”  (OMB,  2005). 

Since  2005,  strategic  sourcing  efforts  have  grown  consistently  within  the  DoD,  with  each 
service  establishing  programs  to  search  for  acquisition  efficiencies  using  this  process. 

Within  the  Air  Force  Civil  Engineer  (CE)  community,  strategic  sourcing  was 
identified  as  a  key  component  of  CE  Transformation  efforts  announced  in  2008  (Eulberg, 
2005).  The  first  enterprise-wide  strategic  sourcing  contract  to  supply  Light  Emitting 
Diode  (LED)  airfield  lighting  on  all  Air  Force  Installations  was  announced  in  2011. 
Additional  strategically  sourced  contracts  are  in  development  for  elevator  maintenance, 
flooring  maintenance,  and  protective  coatings  requirements  (Burt,  2011). 

Currently,  strategic  sourcing  efforts  in  the  Air  Force  are  being  executed  by  the 
Enterprise  Sourcing  Group  (ESG).  Organizationally,  the  ESG  is  divided  into  multiple 
cross-functional  teams  that  manage  strategic  sourcing  efforts  for  each  of  the  eight 
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commodity  groupings.  The  eight  commodity  groupings  are  Information  Technology, 
Medical  Service,  Furnishings,  Force  Protection,  Office  Supplies,  Knowledge-Based 
Services,  and  Civil  Engineering.  This  research  will  focus  primarily  on  the  Civil 
Engineering  Commodity  Council  (CECC). 

The  CECC  consists  of  a  diverse  group  of  members  from  the  ESG,  the  Air  Force 
Civil  Engineer  Center  (AFCEC),  and  the  Air  Staff.  Because  the  CECC  is  not  staffed  to 
provide  a  strategic  sourcing  solution  for  each  of  the  thousands  of  commodities  and 
services  procured  by  the  CE  community,  a  process  known  as  an  opportunity  assessment 
has  been  developed  to  prioritize  opportunities  for  which  strategic  sourcing  solutions  will 
be  implemented.  In  order  to  conduct  this  assessment,  the  CECC  has  been  relying  on 
broadly  accepted  method  of  analyzing  expenditures  to  determine  the  best  opportunities 
for  strategic  sourcing.  This  method,  known  as  a  spend  analysis,  was  developed  to  divide 
procurement  items  into  broad  categories  that  determined  the  optimal  strategy  for  a 
strategic  sourcing  solution. 

Due  to  the  fact  that  the  spend  analysis  method  was  developed  for  use  in  the 
private  sector,  it  focuses  almost  exclusively  on  expenditure  data  without  regard  to 
additional  organizational  objectives.  While  this  may  work  well  for  private  organizations, 
public  sector  organizations,  particularly  the  DoD,  have  a  diverse  set  of  organizational 
values  that  oftentimes  run  counter  to  simply  spending  the  least  amount  possible.  For 
example,  the  stated  objectives  of  the  CECC  listed  in  the  organization’s  charter  document 
are  as  follows  (Civil  Engineering  Commodity  Council,  2010): 

•  Create  enterprise-wide  supplies  and  services  sourcing  strategies 
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•  Create  and  maintain  strategic  supplier  relationships 

•  Drive  commonality  and  standardization  of  requirements 

•  Minimize  supply  chain  cost  through  integration/collaboration 

•  Reduce  procurement  processing  times 

•  Minimize  duplication  of  effort 

•  Lower  total  cost  of  ownership 

•  Leverage  forecasting  data  through  collaboration 

While  minimizing  costs  is  clearly  an  important  part  of  the  CECC’s  mission,  other 
objectives,  such  as  standardization  of  requirements,  are  also  important  aspects  of  the 
organizational  objectives.  Because  of  this  fact,  a  new  method  of  opportunity  assessment 
that  is  capable  of  addressing  the  full  range  of  organizational  objectives  is  needed. 

Problem  Statement 

Current  opportunity  assessment  tools  based  on  private-sector  strategic  sourcing 
efforts  focus  on  developing  strategies  for  implementing  strategic  sourcing  solutions. 
While  these  models  have  proven  to  be  effective  when  used  in  private  industry,  they  fail  to 
provide  a  framework  for  evaluating  the  potential  for  strategic  sourcing  opportunities  to 
achieve  the  full  range  of  organizational  objectives  present  in  a  public  sector  organization. 
A  new  method  of  opportunity  assessment  is  needed  that  can  evaluate  opportunities  based 
on  the  broad  range  of  objectives  of  a  public  sector  strategic  sourcing  program. 
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Research  Questions 

Given  the  importance  of  the  strategic  sourcing  efforts  in  the  CECC  to  save  the 
maximum  amount  of  money  across  the  Air  Force  in  the  shortest  time-frame,  this  research 
is  primarily  focused  on  developing  a  method  of  prioritization  for  existing  service  and 
commodity  contract  areas  that  accounts  for  potential  savings,  costs,  and  efficiencies  of 
strategic  sourcing  efforts  in  the  CECC  according  to  the  objectives  of  Air  Force 
leadership.  The  intent  of  this  research  project  is  to  produce  a  model  that  is  immediately 
usable  by  the  CECC  to  assess  opportunities  for  strategic  sourcing  efforts  given  available 
data  and  resources.  In  order  to  accomplish  this  goal,  the  following  investigative 
questions  will  be  examined: 


•  What  are  the  objectives  that  Air  Force  leadership  believes  strategic  sourcing 
should  accomplish? 

•  What  are  the  relative  priorities  of  those  objectives? 

•  What  variables  predict  potential  efficiencies  in  a  service  or  commodity 
contract  areas? 

•  Can  the  variables  mentioned  above  be  accurately  measured  with  existing  data 
sets  and  current  data  collection  efforts? 

•  What  is  the  model  that  accurately  balances  all  objectives  according  to 
leadership  priorities  that  predicts  progress  toward  strategic  sourcing  goals? 


Methodology 

This  research  project  uses  Value  Focused  Thinking  (VFT)  as  the  decision  analysis 
methodology  for  creating  an  opportunity  assessment  decision  model  for  the  CECC.  This 
model  consists  of  a  hierarchy  of  the  full  spectrum  of  values  important  to  the  CECC  and 
relevant  to  the  opportunity  assessment  problem  as  well  as  a  mathematical  function  that 
converts  an  alternative’s  relevant  data  into  units  of  value  used  to  compare  alternatives. 
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Once  the  VFT-based  model  is  developed,  a  pilot  study  will  be  conducted  using  the  model 
to  analyze  a  small  group  of  strategic  sourcing  opportunities.  The  results  of  the  analysis 
will  be  compared  with  results  of  the  existing  opportunity  assessment  process  in  use  by  the 
CECC  in  order  to  determine  the  impact  of  the  additional  values  on  the  results  of  the 
opportunity  assessment  process.  Where  possible,  data  used  in  the  CECC’s  current 
analysis  of  opportunities  will  be  used  to  evaluate  the  same  alternatives  using  the  new 
model.  Data  requirements  driven  by  the  addition  of  new  values  to  the  opportunity 
assessment  process  will  be  generated  using  a  series  of  interviews  with  subject  matter 
experts  of  the  services  or  commodities  being  evaluated. 

Overview 

This  document  is  arranged  in  five  separate  chapters.  The  following  chapter 
contains  a  review  of  pertinent  literature  that  relates  to  the  subject  of  strategic  sourcing. 
Chapter  III  contains  a  detailed  discussion  on  the  methodologies  used  in  the  generation  of 
results  of  this  research  project.  Chapter  IV  provides  a  detailed  description  of  the  results 
of  the  research  effort  and  an  analysis  of  the  impact  and  significance  of  the  results. 

Finally,  Chapter  V  provides  a  summary  and  a  list  of  conclusions  of  the  project. 
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II.  Literature  Review 


The  purpose  of  this  chapter  is  to  provide  an  overview  of  existing  research  relevant 
to  this  project.  This  chapter  begins  with  a  summary  of  the  background  of  the  strategic 
sourcing  concept,  explores  efforts  within  the  Department  of  Defense  (DoD)  to  implement 
strategic  sourcing,  explains  current  policies  and  procedures  within  the  Air  Force  strategic 
sourcing  organizations,  and  concludes  with  a  background  of  the  multiple  criteria  decision 
analysis  tool  known  as  VFT. 

Strategic  Sourcing 

Over  the  past  35  years,  studies  of  effective  business  procurement  strategies  and 
best  practices  have  developed  the  currently  understood  concept  of  strategic  sourcing. 
Because  the  concept  developed  from  observation  of  practical  results,  strategic  sourcing  is 
less  a  concrete  set  of  methods  for  achieving  supply  excellence  and  more  a  collection  of 
related  ideas  that  are  accepted  as  important  in  formulating  an  effective,  competitive 
supply  strategy.  The  following  paragraphs  provide  a  brief  summary  of  pertinent  research 
that  has  identified  trends  and  concepts  that  have  formed  the  core  of  what  has  come  to  be 
known  as  strategic  sourcing. 

The  idea  of  taking  a  more  strategic  approach  to  purchasing  and  procurement  in 
business  developed  as  an  area  of  research  interest  in  the  early  1980s.  Adamson  (1980) 
summarized  the  emerging  concepts  of  integrating  corporate  strategy  into  supply  and 
purchasing  plans.  He  also  addressed  several  basic  conceptual  questions  identifying  both 
the  need  for  long-range  corporate  planning  and  several  methodologies  for  accomplishing 
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it.  He  proposed  several  methods,  including  linear  programming  and  dynamic 
programming  for  closely  linking  corporate  objectives  to  decisions  made  in  the  planning 
process  (Adamson,  1980). 

Kraljic  (1983)  created  several  practical  models  for  clarifying  the  strategic 
environment  in  which  purchasing  decisions  were  being  made.  Based  on  the  concept  of 
assessing  and  managing  risk  among  various  suppliers  in  an  increasingly  global 
marketplace,  Kraljic’s  purchasing  model,  shown  below  in  Figure  1,  notionally  divides 
supplies  into  four  categories  based  upon  both  the  importance  of  the  asset  to  corporate 
objectives  and  the  complexity  of  the  market  in  which  the  product  is  available  (Kraljic, 
1983). 


HIGH  t 


Importance 

of 

Purchasing 


LOW 


Figure  1.  Kraljic’s  Purchasing  Model  (1983) 


Materials  Management 

Supply  Management 

Focus:  Leverage  items 

Focus:  Strategic  items 

Criteria:  Cost  and  flow  of 
materials 

Criteria:  Long-term 
availability 

Purchasing  Management 

Sourcing  Management 

Focus:  Noncritical  items 

Focus:  Bottleneck  items 

Criteria:  Efficiency 

Criteria:  Cost  and  reliable 
short-term  sourcing 

- ► 

LOW  HIGH 

Complexity  of  Market 


Each  of  the  four  groups  has  a  distinct  focus  as  well  as  different  criteria  by  which 
to  evaluate  opportunities  found  in  each  category.  This  method  provides  a  systematic, 
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comprehensive  analysis  of  all  procurement  items,  and  helps  the  user  develop  sourcing 
strategies  tailored  to  the  needs  and  environment  of  each  individual  requirement.  For 
example,  items  with  high  importance  but  little  market  complexity  lend  themselves  to  a 
purchasing  strategy  that  seeks  to  leverage  the  purchasing  power  of  the  organization  to 
achieve  lower  costs  (Kraljic,  1983).  This  strategy  in  particular  will  be  discussed  later  in 
this  chapter. 

As  globalization  increased  competition  in  the  marketplace  in  the  late  1980s, 
Speckman  (1988)  highlighted  the  emerging  trend  in  industry  for  the  integration  of 
companies  with  their  suppliers.  He  proposed  a  method  of  evaluating  and  selecting 
potential  suppliers  for  strategic,  integrated  relationships  based  on  the  experience  and 
capability  of  the  supplier  as  well  as  the  volume  and  importance  of  the  commodity  being 
purchased  (Speckman,  1988).  This  concept  has  formed  one  of  the  central  ideas  of 
strategic  sourcing,  extending  the  idea  of  integrating  long-term  corporate  planning  into  the 
supply  strategy  to  develop  buyer- supplier  relationships. 

Currently,  the  concept  of  strategic  sourcing  has  become  an  accepted  best  practice 
in  the  discipline  of  supply  chain  management,  and  variations  of  its  concepts  can  be  found 
within  the  pages  of  virtually  all  texts  on  purchasing  and  supply  chain  management. 

While  it  has  grown  to  be  synonymous  with  making  good  procurement  decisions,  there  are 
several  concepts  that  are  central  to  the  idea.  Johnson,  Leenders,  and  Flynn  (2006)  define 
strategic  sourcing  as  focusing  “on  long-term  supplier  relation  and  commodity  plans  with 
the  objectives  of  identifying  opportunities  in  areas  such  as  cost  reduction,  new 
technology  advancements,  and  supply  market  trends”.  This  definition  highlights  several 
of  the  central  ideas  of  strategic  sourcing:  that  it  involves  deliberate  well-thought  out 
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planning,  that  it  involves  developing  inter-connected  relationships  between  buyer  and 
supplier,  and  that  it  can  be  used  to  further  organizational  objectives. 

While  the  literature  relevant  to  strategic  sourcing  clearly  defines  what  it  is  and 
explains  strategies  for  strategically  sourcing  the  goods  and  services  an  organization 
procures,  very  little  is  written  regarding  decision  models  related  to  strategic  sourcing. 
Talluri  and  Narasimhan  (2004)  presented  a  decision  model  related  to  the  selection  of  an 
organization’s  suppliers  using  Data  Envelopment  Analysis  (DEA).  Other  published 
methodologies,  like  Kraljic’s  method  described  previously,  focus  on  finding  the  best 
solution  for  a  strategic  sourcing  opportunity,  not  on  identifying  what  the  best 
opportunities  are.  While  no  reason  for  this  is  stated  in  the  literature,  it  is  reasonable  to 
conclude  that  researchers  generally  assume  that  an  organization  will  be  making  a 
strategic  sourcing  decision  on  every  service  and  commodity  purchased  enterprise-wide. 
Because  no  methodology  has  been  developed  to  address  the  unique  opportunity 
assessment  problem  in  the  DoD,  organizations  have  been  left  with  developing  their  own 
methods  for  opportunity  assessment. 

Strategic  Sourcing  in  the  Department  of  Defense 

While  the  concept  of  strategic  sourcing  has  been  widely  used  in  the  private  sector, 
little  effort  was  made  to  implement  any  of  the  best  practices  in  the  government  until  the 
early  2000s.  In  2002,  the  Government  Accountability  Office  (GAO)  released  a  report 
highlighting  the  potential  for  savings  in  government  acquisition  programs  based  on  case 
studies  of  six  companies  that  had  implemented  strategic  sourcing  programs.  The  GAO 
(2002)  found  that  implementing  strategic  procurement  practices  resulted  in  lower  cost, 
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higher  quality  products.  The  report  cited  four  common  components  that  contributed  to 
success  in  each  of  the  six  organizations:  an  organizational  commitment  to  a  strategic 
approach;  improved  knowledge  of  spending  patterns;  a  sufficient  supporting  structure, 
processes,  and  roles;  and  a  leadership  focus  including  communication  of  metrics  to 
subordinates  (GAO,  2002). 

In  response  to  this  study  and  the  2002  Defense  Authorization  Act,  the  DoD  began 
a  pilot  program  aimed  at  implementing  a  comprehensive  spend  analysis  of  enterprise¬ 
wide  expenditures  on  service  acquisitions.  According  to  a  follow-up  study  by  the  GAO 
(2003),  while  the  program  did  attempt  to  implement  a  one-time  spend  analysis  of  some 
service  contract  areas,  it  failed  to  provide  a  repeatable  and  comprehensive  examination  of 
enterprise-wide  service  contract  spending.  DoD  cited  several  reasons  for  this  problem, 
including  decentralized  procurement  practices  and  the  multitude  of  disparate  financial 
programs  that  are  used  to  track  expenditures  in  various  DoD  organizations  (GAO,  2003). 

The  Office  of  Management  and  Budget  subsequently  released  a  memorandum 
requiring  all  federal  agencies  to  take  steps  to  implement  a  strategic  sourcing  program 
with  the  overall  goal  of  reducing  the  cost  of  government  operations  (OMB,  2005).  This 
memo  defined  strategic  sourcing  as  “the  collaborative  and  structured  process  of  critically 
analyzing  an  organization’s  spending  and  using  this  information  to  make  business 
decisions  about  acquiring  commodities  and  services  more  effectively  and  efficiently” 
(OMB,  2005).  In  response,  strategic  sourcing  efforts  have  grown  consistently  within  the 
DoD,  with  each  service  establishing  programs  to  search  for  acquisition  efficiencies  using 
this  process. 
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Air  Force  Strategic  Sourcing 

Prior  to  the  release  of  the  OMB  directive  for  all  federal  departments  to  utilize 
strategic  sourcing,  the  Air  Force  began  its  strategic  sourcing  program  with  the 
formulation  of  the  Information  Technology  Commodity  Council  in  2003.  This 
organization  was  successful  in  demonstrating  the  capability  of  strategic  sourcing  to 
improve  efficiencies  with  the  acquisition  of  commodities  (IT  Commodity  Council,  2013). 
In  2009,  the  Air  Force  launched  an  expansion  of  the  Commodity  Council  concept  with 
the  establishment  of  the  Enterprise  Sourcing  Group  at  Wright  Patterson  Air  Force  Base, 
Ohio.  This  new  organization  was  established  in  2010  and  was  charged  with  the 
formulation  and  oversight  of  the  all  Air  Force  commodity  councils,  including  civil 
engineering,  force  protection,  furnishings,  information  technology,  knowledge-based 
services,  medical,  and  office  supplies  (Enterprise  Sourcing  Group,  2013). 

The  Civil  Engineer  Commodity  Council  (CECC)  serves  as  the  primary  hub  of 
strategic  sourcing  actions  for  the  Civil  Engineering  (CE)  community.  Unique  in 
organizational  structure,  the  CECC  is  charged  with  managing  strategic  sourcing  solutions 
for  the  CE  community  while  not  actually  being  inside  the  CE  chain  of  command.  The 
CECC  primarily  interfaces  with  the  Air  Force  Civil  Engineer  Center  (AFCEC)  to  obtain 
feedback  and  information  in  order  to  meet  the  evolving  needs  of  Air  Force  CE  (Civil 
Engineering  Commodity  Council,  2013). 

The  CECC’s  strategic  sourcing  process  is  graphically  detailed  in  Figure  2. 
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■  Rollout/resource  strategy 

■  Issue  policy  and/or 

■  Execute  demand  management 


■  Identify  qualified  suppliers 

■  Understand  industry  trends 


■  Define  requirements 

■  Standardize  requirements 


and  cost  structure 


Customer 


and/or 

■  Follow  acquisition  process 


Warfighter 


■  Consolidate  results  and  define  strategy 

■  Obtain  approval  for  strategy 


Taxpayer 


Figure  2.  The  Air  Force  Strategic  Sourcing  Process  Model  (ESG,  2011) 

As  Figure  2  shows,  the  strategic  sourcing  process  begins  with  an  opportunity 
assessment,  where  potential  areas  of  commodity  or  service  contracts  are  evaluated  for  the 
potential  to  reap  savings  for  the  Air  Force.  Potential  contract  areas  are  then  undergo  a 
review  process  that  includes  an  evaluation  of  existing  contract  processes,  an  evaluation  of 
the  commercial  market  for  the  commodity  or  service,  and  a  definition  of  the  user’s 
requirements  for  the  product  or  service.  This  information  is  then  used  to  generate  a 
sourcing  strategy  that  will  result  in  the  optimal  contract  solution  for  the  user  on  an 
enterprise-wide  level.  Once  this  sourcing  strategy  is  approved,  the  ESG  competes, 
awards,  and  monitors  the  contract  throughout  its  lifecycle. 

Currently,  the  opportunity  assessment  phase  of  the  strategic  sourcing  process  is 
being  executed  via  a  spend  analysis  similar  to  Kraljic’s  method.  Historical  spend  data  are 
captured  primarily  from  two  databases,  the  Contracting  Business  Intelligence  System 
(CBIS)  and  the  Commander’s  Resource  Integration  System  (CRIS)  and  then  analyzed  to 
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identify  the  commodities  and  services  where  the  most  money  is  being  spent.  These  high 
spend  areas  are  then  more  closely  examined  for  potential  opportunities  for  efficiencies. 
This  method  has  resulted  in  one  successful  strategic  sourcing  contract,  Light  Emitting 
Diode  (LED)  taxiway  lighting,  being  executed  by  the  Enterprise  Sourcing  Group  since 
2010. 

Value  Focused  Thinking 

Value-Focused  Thinking  (VFT)  is  a  Multiple  Criteria  Decision  Analysis  (MCDA) 
method  developed  by  Keeney  (1992)  that  is  based  on  broad  organizational  values  or 
objectives.  The  VFT  process  requires  the  decision-maker  to  first  identify  his  or  her 
values  that  are  relevant  to  the  decision  that  needs  to  be  made.  This  contrasts  from 
traditional  alternative-focused  decision  analysis  methods  in  that  it  focuses  on  developing 
alternatives  and  evaluation  criteria  after  the  organizational  goals  and  objectives  have  been 
identified.  Alternative-focused  decision  methods  suffer  from  being  focused  on  merely 
finding  ways  to  differentiate  each  alternative,  rather  than  evaluating  alternatives  based  on 
the  strategic  objectives  of  the  decision-maker.  This  reactionary  method  also  stovepipes 
decision  makers  into  only  considering  alternatives  presented  at  the  outset  of  the  problem¬ 
solving  process  (Keeney,  1992).  Alternatively,  VFT  seeks  to  fully  understand  the 
underlying  objectives  behind  the  decision,  leading  the  decision-maker  down  a  path  of 
greater  understanding  that  results  in  the  potential  for  alternatives  to  be  developed  based  in 
the  insight  gained  during  the  value  identification  process. 

In  addition  to  increased  understanding  of  the  values  important  to  the  decision¬ 
maker,  the  VFT  method  results  in  decision  models  that  are  flexible  to  changes  in 
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alternatives  and/or  criteria  without  the  need  to  completely  revise  the  model.  This  is 
particularly  useful  for  an  application  such  as  strategic  sourcing  opportunity  assessments, 
where  there  are  a  multitude  of  ever-changing  alternatives.  Other  MCDA  methods, 
namely  Saaty’s  Analytical  Hierarchy  Process,  require  decision  models  to  be  completely 
re-generated  if  any  change  is  made  to  the  alternatives  or  criteria  to  be  evaluated  (Saaty, 
1980).  The  insight  gained  during  the  VFT  process  and  the  flexibility  of  the  resulting 
decision  model  makes  VFT  ideally  suited  for  use  in  conducting  an  opportunity 
assessment  for  public  sector  organizations.  The  following  paragraphs  will  detail  the  eight 
specific  steps  of  the  VFT  decision  making  process  as  set  forth  by  Kirkwood  (1997)  and 
shown  graphically  in  Figure  3. 


Figure  3.  Kirkwood’s  eight-step  VFT  process  (1997) 

Step  1:  Identify  Decision 

The  first  step  in  the  VFT  process  is  to  identify  the  decision  that  needs  to  be  made. 
While  this  step  may  seem  obvious,  failure  to  fully  understand  the  context  and  intent  of 
the  decision  to  be  made  will  result  in  a  model  that  does  not  necessarily  fit  the  actual 
decision  space. 
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Step  2:  Structure  Objectives 

As  the  VFT  name  implies,  clearly  identifying  the  values  or  objectives  that  are 
applicable  to  the  decision  is  essential  to  the  decision  making  process.  One  method  of 
accomplishing  this  is  by  creating  a  value  hierarchy.  A  value  hierarchy  is  a  graphical 
depiction  of  the  full  range  of  objectives  relevant  to  the  decision  in  question.  The 
hierarchy  is  arranged  with  a  primary,  or  fundamental,  objective  at  the  top,  with 
subsequent  tiers  of  subordinate  objectives  listed  below  (Keeney,  1992).  Figure  4 
provides  a  notional  diagram  of  a  typical  hierarchy  structure. 


Figure  4.  Notional  Diagram  of  a  Value  Hierarchy 

Kirkwood  (1997)  identifies  five  desirable  properties  of  a  value  hierarchy.  First,  it 
should  be  complete.  This  means  it  should  cover  the  full  range  of  values  relevant  to  the 
decision.  In  other  words,  no  part  of  the  fundamental  objective  is  being  ignored  and  it  can 
be  fully  evaluated  based  on  the  subordinate  tier  objectives  present  in  the  hierarchy 
structure.  The  second  desirable  property  is  nonredundancy.  Specific  sub-objectives 
should  not  be  found  within  multiple  tiers  as  this  will  lead  to  double-counting  of  the 
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objective  when  it  is  evaluated.  The  third  desirable  property  is  independence.  This  means 
each  objective  must  be  able  to  be  evaluated  in  a  manner  that  is  comparable  to  the  other 
objectives.  While  it  is  possible  for  objectives  to  require  that  dissimilar  metrics  be 
compared  (e.g.,  money  vs.  intangible  tradeoffs  such  as  environmental  quality),  a  means 
must  be  available  to  convert  the  metrics  into  a  common  system  of  measure.  This  is 
accomplished  in  VFT  models  through  the  use  of  Single  Dimensional  Value  Functions 
(SDVFs).  The  fourth  value  hierarchy  quality  that  is  needed  is  operability.  Operability 
means  simply  that  the  objective  is  clearly  and  explicitly  defined  and  able  to  be 
understood  within  the  decision  context.  Finally,  small  size  is  the  fifth  desirable  property. 
Hierarchies  should  be  as  small  as  possible  to  meet  the  other  four  desirable  qualities  and 
reduce  the  burden  of  data  gathering  and  calculations  on  the  part  of  the  decision  maker 
(Kirkwood,  1997). 

Step  3:  Develop  Evaluation  Measures 

Once  the  values  have  been  identified,  a  means  must  be  developed  to  measure  the 
relative  merits  of  each  alternative.  These  measures  are  the  bridge  that  connects  the 
desired  objectives  with  the  data  available  to  the  decision  maker.  There  are  three  primary 
types  of  evaluation  measures:  natural,  proxy,  and  constructed  measures.  Natural 
evaluation  measures  lend  themselves  to  direct,  quantitative  measurement.  As  the  name 
implies,  natural  objectives  are  intuitive  and  require  that  the  objective  both  be 
quantitatively  measurable  and  that  measurement  is  possible  for  the  decision  maker. 
Natural  evaluation  measures  are  the  preferred  method  of  evaluating  alternatives  as  they 
provide  the  most  objective  measurement  to  the  process.  Less  desirable  than  natural 
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attributes,  constructed  attributes  are  used  when  no  quantifiable  measurement  is  possible 
related  to  a  specific  objective.  Typically,  these  are  accomplished  by  formulating  a  scale 
that  an  expert  uses  to  assign  a  “score”  directly  evaluating  an  alternative  based  on  its 
potential  to  further  the  objective  in  question.  Finally,  the  proxy  attribute  is  the  least 
desirable  method  of  evaluating  alternatives.  The  proxy  evaluation  measure  is  for 
objectives  that  do  not  lend  themselves  to  direct  measurement  by  either  a  natural  or 
constructed  measure,  but  for  which  there  is  data  available  that  indirectly  measures  the 
alternative’s  contribution  toward  the  objective  in  question  (Keeney,  1992). 

Step  4:  Develop  Alternatives 

Alternative  creation  is  a  step  in  the  decision  analysis  process  that  is  unique  to 
VFT.  In  accomplishing  the  previous  three  process  steps,  the  decision  maker  gains  a 
significant  amount  of  insight  into  the  objectives  and  motivations  that  frame  the  decision 
context.  Using  this  insight,  a  decision  maker  is  well-prepared  to  evaluate  the  known 
alternatives  and  generate  new  hybrid  alternatives  that  seems  to  best  fit  the  objectives  that 
were  identified  (Keeney,  1992). 

Step  5:  Create  Value  Functions 

Single  Dimensional  Value  Functions  (SDVF)  are  mathematical  representations  of 
the  relationship  between  an  alternative’s  data  point  for  a  given  evaluation  measure  and  its 
associated  score  that  indicates  a  positive  impact  on  the  fundamental  objective.  Each 
evaluation  measure  must  have  this  relationship  clearly  established  to  create  the  overall 
value  function  that  will  identify  the  preferred  alternative.  This  is  accomplished  by 
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converting  data  for  all  evaluation  measures  into  units  of  value  via  a  SDVF.  SDVFs  can 
be  linear,  curvilinear,  or  discrete,  but  must  either  increase  or  decrease  in  value 
mono  tonic  ally.  That  is,  the  slope  must  always  be  positive  or  negative  over  the  entire 
range  of  possible  values.  SDVFs  provide  the  value  hierarchy  with  the  independence 
necessary  to  allow  for  comparison  of  dissimilar  objectives.  (Kirkwood,  1997). 
Graphical  examples  of  different  types  of  SDVFs  are  provided  in  Figure  5. 


Categorical 


Linear 


Piecewise  Linear 


Figure  5.  Single  Dimensional  Value  Function  Examples 

Step  6:  Develop  Weights 

The  next  component  of  the  value  function  is  the  weight  factor  that  is  assigned  to 
each  evaluation  measure.  Weighting  factors  are  used  to  model  the  relative  importance  of 
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each  evaluation  measure  as  it  relates  to  the  fundamental  objective.  There  are  two  primary 
methods  for  determining  evaluation  measure  weights,  the  direct  method  and  the  swing 
method.  Direct  weighting  is  accomplished  by  having  the  decision  maker  directly  assign 
relative  importance  to  each  of  the  objectives  in  the  objective  hierarchy  by  each  branch 
with  a  tier.  Typically,  this  is  accomplished  with  a  top-down  approach,  where  weights  are 
first  assigned  to  the  top  tiers  of  the  hierarchy,  and  then  weights  are  developed  for  the 
subordinate  set  of  objectives  as  a  group.  This  process  leads  to  the  development  of  local 
and  global  weights  for  each  objective.  Local  weights  refer  to  the  weight  assigned  under 
the  parent  objective,  whereas  the  global  weight  factors  in  the  weighting  of  the  parent  tiers 
into  the  overall  weight  of  the  sub-objective  (Shoviak,  2001). 

Weights  are  assigned  as  percentages,  where  a  value  of  1  refers  to  the  entirety  of 
available  weight.  One  method  of  conceptualizing  this  weighting  system,  called  the  ‘TOO 
coin  method,”  is  for  decision  makers  to  assume  they  have  100  coins  to  distribute  among 
the  objectives  that  represent  units  of  weighting.  Once  all  100  coins  have  been  distributed 
among  the  objectives,  percentage  weights  have  been  determined.  For  example,  if  fifty 
coins  were  given  to  a  particular  objective,  it  would  receive  a  weight  of  0.5,  and  would 
correspond  to  the  concept  that  that  particular  objective  is  worth  half  of  the  entire  weight 
in  the  hierarchy  (Jurk,  2002). 

Swing  weighting  is  the  weighting  method  proposed  by  Keeney,  and  refers  to 
weights  being  determined  by  evaluating  the  effects  of  weights  assigned  and  evaluating 
those  effects  in  an  iterative  process.  Typically,  this  is  accomplished  via  a  pair-wise 
comparison  between  two  objectives  at  a  time.  The  decision  maker  develops  a  micro¬ 
version  of  the  value  equation  using  only  the  two  objectives  that  are  being  considered  in 
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this  particular  comparison.  Assigning  notional  scores  and  seeing  the  outcome  of  the 
equation  gives  the  decision  maker  a  less  subjective  picture  of  the  actual  relative 
importance  of  objective,  particularly  when  the  objectives  are  quantified  in  a  dissimilar 
manner  (Keeney,  1992). 

Step  7:  Determine  Overall  Values  for  Alternatives 

Once  SDVFs,  objective  weights,  and  data  have  been  gathered,  the  value  function 
can  be  created  according  to  Equation  2.1  (Kirkwood,  1997): 

v(*)  =  Z"=i  WiVfXi)  (2.1) 

Where  v  represents  the  total  value  score  for  the  alternative,  w,  represents  the  weight 
factor  for  the  zth  objective,  v,  represents  the  SDVF  for  the  zth  objective,  and  xt  represents 
the  data  input  for  the  zth  objective.  This  function  generates  an  overall  value  score  for 
each  alternative  that  can  be  used  for  comparison. 

Step  8:  Select  Alternatives 

Once  each  of  the  alternatives  has  received  an  aggregate  value  score,  a  basis  for 
comparison  exists  that  can  be  used  to  select  the  most  attractive  alternative.  While  this  is 
a  straightforward  process,  simple  steps  can  be  taken  to  double  check  the  model  itself. 
Chambal  recommends  conducting  a  sensitivity  analysis  on  the  higher  tiers  of  the  value 
hierarchy  in  order  to  explore  how  varying  the  weights  affects  the  decisions  recommended 
by  the  value  function.  This  is  accomplished  by  altering  the  global  weight  of  a  single 
objective,  while  maintaining  proportional  weights  across  the  remaining  objectives.  This 
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analysis  is  useful  in  identifying  alternatives  could  potentially  look  more  attractive  given  a 
small  adjustment  in  the  weight  factors  (Chambal,  Weir,  Yucel,  &  Gutman,  2011). 

Chapter  Summary 

While  strategic  sourcing  has  yielded  significant  results  in  both  the  public  and 
private  sector,  DOD  has  struggled  with  implementing  a  thorough,  enterprise-wide  spend 
analysis  on  its  expenditures.  This  is  due  largely  to  decentralized  management  practices 
that  have  made  gathering  the  data  necessary  to  accomplish  the  analysis  extremely 
difficult.  As  a  result,  a  thorough  and  comprehensive  method  of  accomplishing  an 
opportunity  assessment  in  the  CECC  has  not  been  developed.  This  has  left  the  CECC 
with  little  insight  into  which  of  the  multitude  of  contract  areas  in  which  to  begin 
investigating  a  strategically  sourced  solution.  VFT  can  be  used  to  develop  a  decision 
support  tool  that  can  be  used  to  effectively  conduct  a  systematic,  comprehensive,  and 
objective  opportunity  assessment  of  all  possible  contract  areas.  The  next  chapter  will 
provide  a  detailed  explanation  of  the  specific  methodology  applied  in  this  research 
project. 
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III.  Methodology 


The  purpose  of  this  chapter  is  to  explain  in  detail  the  methodology  used  in  this 
research  effort.  Specifically,  it  provides  details  of  the  application  of  Value  Focused 
Thinking  (VFT)  to  the  opportunity  assessment  Multiple  Criteria  Decision  Analysis 
(MCDA)  problem  faced  in  the  Civil  Engineer  Commodity  Council  (CECC).  This  section 
begins  with  an  overview  of  the  model  development  process,  and  then  explains  in  detail 
the  specifics  of  the  eight  step  process  used  in  this  research  project,  including  the  data 
gathering  process.  The  decision  model  developed  during  this  research  project  is 
described  in  this  chapter,  as  explaining  the  results  of  certain  process  steps  is  necessary  to 
adequately  explain  the  methods  used  in  subsequent  steps.  The  model  will  be  fully 
presented  again  in  Chapter  IV  with  the  remainder  of  the  results. 

The  VFT  Process 

VFT  differs  from  other  MCDA  methods  in  that  it  evaluates  alternatives  based 
upon  the  values  of  the  decision-maker,  and  not  merely  based  on  characteristics  that 
differentiate  known  alternatives.  Because  of  this,  the  VFT  process  requires  an  additional 
set  of  analyses  in  order  to  define  the  values  of  the  decision-maker.  This  is  of  particular 
importance  in  the  Civil  Engineering  Commodity  Council  (CECC)  opportunity  assessment 
decision  context  due  to  the  wide  range  of  strategic  objectives  important  to  Air  Force  Civil 
Engineering  (CE)  acquisitions  that  the  current  opportunity  assessment  methods  fail  to 
address. 
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To  accurately  capture  and  understand  the  strategic  objectives  relevant  to  this 
decision-making  process,  Kirkwood’s  eight-step  VFT  model  (1997)  was  used  as  the 
overarching  methodology  in  this  research  project.  Figure  6  below  provides  an  overview 
of  the  eight-step  VFT  process  along  with  a  diagram  of  the  process  inputs  used.  This 
diagram  depicts  the  sequence  of  the  VFT  methodology  as  well  as  broad  categories  of 
inputs  to  various  steps  of  the  decision  making  process.  A  combination  of  expert  opinion, 
published  official  documents,  and  quantitative  data  was  used  throughout  the  model 
building  process  to  assign  value  to  alternatives  based  on  objectives.  The  following 
sections  of  this  chapter  explain  in  detail  the  process  steps  executed  during  this  research 
project. 


Figure  6.  VFT  Process  Diagram  with  Inputs 
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Step  1.  Problem  Identification 

The  opinion  that  the  right  strategic  sourcing  opportunities  were  not  being 
evaluated  was  initially  identified  during  informal  conversations  with  engineers  at  Air 
Force  Materiel  Command  (AFMC).  Further  discussions  with  engineers  at  the  Major 
Command  and  Field  Operating  Agency  level  mirrored  the  comments  made  at  AFMC. 
Based  on  these  discussions,  a  meeting  was  held  with  members  of  the  Enterprise  Sourcing 
Group  (ESG),  the  organization  responsible  for  the  planning  and  execution  of  the  Air 
Force’s  strategic  sourcing  program.  During  this  meeting,  ESG  staff  indicated  that  a 
problem  existed  regarding  the  opportunity  assessment  phase  of  the  strategic  sourcing 
process.  While  a  process  for  conducting  opportunity  assessments  had  been  in  use  for  two 
years,  it  failed  to  accurately  capture  the  full  range  of  objectives  important  to  the  Air  Force 
CE  functional  community.  As  a  result,  the  ESG  had  been  struggling  to  deliver  strategic 
sourcing  contracts  and  were  having  difficulty  identifying  new  opportunities  for 
evaluation. 

Based  on  this  information,  this  research  project  was  implemented  to  create  an 
alternative  means  of  opportunity  assessment  that  would  accurately  capture  the  full  range 
of  objectives  relevant  to  the  CECC  and  the  Air  Force  CE  community.  Upon  review  of 
literature  relevant  to  the  problem,  VFT  emerged  as  the  methodology  best  suited  to 
providing  a  solution  to  the  CECC’s  opportunity  assessment  problem.  Once  the  problem 
and  associated  methodology  were  identified,  the  next  step  was  to  define  what  exactly  the 
objectives  of  the  CE  strategic  sourcing  program  were. 


25 


Step  2.  Structure  Objectives 

The  first  step  of  understanding  the  context  in  which  the  opportunity  assessment 
decision  problem  occurs  is  to  identify  the  strategic  objectives  of  the  strategic  sourcing 
program.  While  most  VFT  applications  require  “deep  and  serious  thought”  (Keeney, 
1992),  many  of  the  objectives  of  the  CECC  were  stated  in  the  organization’s  charter 
document.  Eight  objectives  were  developed  jointly  by  the  CE  and  Contracting  functional 
leadership  to  provide  a  broad  vision  of  what  the  CECC  was  meant  to  accomplish.  The 
eight  objectives  contained  in  the  charter  are  as  follows  (Civil  Engineering  Commodity 
Council,  2010): 

1.  Create  enterprise-wide  supplies  and  services  sourcing  strategies 

2.  Create  and  maintain  strategic  supplier  relationships 

3.  Drive  commonality  and  standardization  of  requirements 

4.  Minimize  supply  chain  cost  through  integration/collaboration 

5.  Reduce  procurement  processing  times 

6.  Minimize  duplication  of  effort 

7.  Lower  total  cost  of  ownership 

8.  Leverage  forecasting  data  through  collaboration 

The  first  objective,  to  create  enterprise-wide  sourcing  strategies,  applies  broadly  to  the 
overall  mission  of  the  CECC,  but  does  not  have  any  direct,  measurable  bearing  on  the 
opportunity  assessment  phase.  Three  other  objectives,  numbers  2,  5,  and  8,  apply  to  the 
contracting  process  that  occurs  after  the  opportunity  assessment  has  been  conducted.  The 
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remaining  four  objectives  provide  the  basis  for  the  desirable  elements  of  a  strategic 
sourcing  contract. 

Each  of  these  objectives  fall  under  the  overall  objective  of  strategic  sourcing, 
which  is  “to  make  business  decisions  about  acquiring  commodities  and  services  more 
effectively  and  efficiently”  (OMB,  2005).  Several  of  these  objectives  can  be  further 
divided  into  sub-objectives  in  order  to  accurately  capture  the  various  dimensions  of 
meaning  inherent  in  the  language.  This  relationship  is  shown  in  Figure  7  as  an  objective 
hierarchy.  This  hierarchy  displays  the  association  between  all  objectives,  stated  and 
implicit,  in  the  CECC  charter  document. 


Figure  7.  Objective  Hierarchy  of  CECC  Charter  Strategic  Objectives 
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As  shown  in  Figure  7,  the  objectives  stated  in  the  CECC  charter  can  be  clarified  to  apply 
more  directly  to  the  opportunity  assessment  process  itself.  Based  on  the  CECC  charter 
document,  the  following  list  of  opportunity  assessment  objectives  was  developed: 

1.  Reduce  the  number  of  diverse  solutions  for  a  given  requirement 

2.  Establish  and/or  update  standards  and  guidance  for  a  given  requirement 

3.  Reduce  the  logistics  cost  on  contracts 

4.  Develop  strategic  partnerships  with  suppliers 

5.  Reduce  the  number  of  contracting  agencies  working  on  a  given 
requirement 

6.  Reduce  the  number  of  contracts  needed  for  a  given  requirement 

7.  Lower  first  costs  of  commodities  and  services 

8.  Lower  lifecycle  costs  of  commodities  and  services 

In  addition  to  the  objective  hierarchy  distilled  from  the  CECC  charter  document,  a 
series  of  informal  interviews  with  panels  of  subject  matter  experts  was  used  to  develop 
the  objective  hierarchy  for  this  decision  problem.  This  was  primarily  done  for  two 
reasons:  the  charter  document  was  published  prior  to  the  formation  of  the  current  process 
and  the  ESG  personnel  had  been  operating  since  the  organization  was  formed  without 
knowledge  of  these  objectives.  By  combining  the  approved,  official  CECC  charter 
documents  with  the  current  working  knowledge  of  the  subject  matter  experts,  a  more 
accurate  model  of  the  true  strategic  objectives  of  the  strategic  sourcing  program  was 
developed  that  meets  Parnell’s  (1998)  gold  standard  as  defined  in  Chapter  II. 
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Subject  matter  experts  were  consulted  from  both  the  ESG  and  the  Air  Force  Civil 
Engineer  Center  (AFCEC),  with  data  being  collected  in  two  phases.  First,  the  ESG 
engineers  developed  a  draft  hierarchy  independent  of  input  from  published  official 
documents  and  the  subject  matter  experts  from  AFCEC.  This  was  accomplished  through 
a  brainstorming  process  whereby  potential  objectives  were  written  onto  note  cards.  As 
objectives  were  suggested,  the  cards  were  taped  to  the  wall  of  the  conference  room  and 
arranged  according  to  similar  categories.  As  the  hierarchy  evolved,  three  broad 
categories  of  objectives  emerged:  rate-related  objective  (cost  and  quality),  process 
efficiency-related  objectives,  and  demand  management  objectives.  These  categories  also 
aligned  with  the  current  terminology  in  use  by  the  senior-level  decision-makers  at  the 
ESG,  and  were  therefore  adopted  as  the  tier  1  objectives  with  the  same  terminology  to 
facilitate  ease  of  understanding  with  stakeholders  and  decision-makers.  Through  this 
process,  the  overall  objective  “Support  the  CE  Mission  by  improving  the  efficiency  of  CE 
acquisitions”  was  formulated.  After  this  first  iteration  of  objective  hierarchy  building 
was  completed,  the  draft  hierarchy  was  sent  to  subject  matter  experts  at  AFCEC  for 
review  and  comment.  The  draft  hierarchy  is  provided  in  Appendix  A. 

Upon  receipt  of  the  revisions  and  comments  from  AFCEC  subject  matter  experts, 
the  draft  hierarchy  was  again  presented  to  the  ESG  panel  of  experts  along  with  the 
objectives  obtained  from  review  of  the  CECC  charter  document  presented  in  Figure  7. 
Using  this  information,  the  panel  created  a  revised  objective  hierarchy  that  was  submitted 
back  to  AFCEC  for  approval  and  was  validated  by  CECC  leadership.  The  resulting 
objective  hierarchy  is  displayed  in  Figure  8,  and  definitions  of  each  objective  are 
provided  in  Table  1  and  Table  2. 
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Table  1.  Tier  1  Objectives  and  Definitions 


Objective  Name 

Definition 

Create  Rate  Savings 

This  category  contains  objectives  that  are  associated  with  managing 
the  cost  of  the  service  or  commodity  that  is  being  considered  for 
strategic  sourcing. 

Improve  Processes 

Improve  processes  refers  to  objectives  that  promote  efficiencies  in 
services  and  commodities  by  reducing  the  risk  and  resource  demands 
associated  with  the  acquisitions  process. 

Manage  Demand 

This  category  refers  to  realizing  efficiencies  related  to  increasing  or 
decreasing  demand  for  commodities,  services,  or  resources. 

Table  2.  Tier  2  Objectives  and  Definitions 


Objective  Name 

Definition 

Reduce  first  costs 

Strategic  sourcing  solutions  should  reduce  the  initial  expenses  related 
to  the  acquisition  of  services  and  commodities 

Reduce  O&M  costs 

Strategic  sourcing  solutions  should  reduce  the  Operations  and 
Maintenance  (O&M)  burden  in  both  manpower  and  resource 
requirements  for  services  and  commodities 

Reduce  final  costs 

Strategic  sourcing  solutions  should  reduce  the  final  cost  component  of 
services  and  commodities  acquired.  This  includes,  but  it  not  limited 
to,  disposal,  demolition,  environmental  remediation,  and  other  removal 
costs  associated  with  the  commodity  or  service 

Improve  quality 

Strategic  sourcing  solutions  should  improve  the  quality  of 
commodities  and  services  acquired 

Reduce  processing 
time 

Strategic  sourcing  solutions  should  seek  to  reduce  the  burden  of  the 
acquisition  system  by  both  reducing  the  number  of  contracts,  and  the 
resource  requirements  associated  with  them 

Leverage  new 
technology 

Strategic  sourcing  solutions  should  seek  to  leverage  new  technology, 
where  appropriate  and  feasible,  across  the  Air  Force  that  can  improve 
the  efficiency  of  services  and  commodities 

Streamline 

regulations 

Strategic  sourcing  efforts  should  reduce  the  overall  burden  of  outdated 
and  conflicting  guidance  and  regulations  to  the  acquisitions  system 

Minimize  small 
business  risk 

Strategic  sourcing  contracts  should  support  the  federal  government’s 
philosophy  of  encouraging  free  and  open  competition  between 
vendors,  and  should  strive  to  support  government  small  business 
participation  goals 

Reduce  utility  usage 

Strategic  sourcing  efforts  should  seek  to  reduce  the  utility  demand 
associated  with  the  services  and  commodities  that  are  evaluated 

Improve 

standardization 

Strategic  sourcing  should  exploit  the  potential  for  efficiencies  by 
increasing  demand  through  standardizing  solutions  to  common 
requirements  where  feasible 

Introduce  COLs 

Strategic  sourcing  efforts  should  manage  demand  for  services  through 
the  creation  and  implementation  of  Common  Output  Levels  (COLs) 
that  standardize  requirements  for  common  services  across  the 
enterprise 
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Step  3:  Develop  Evaluation  Measures 

Once  the  objective  hierarchy  had  been  established,  evaluation  measures  were 
developed  for  each  tier-two  objective  to  quantify  the  units  of  value  for  each  alternative  in 
furthering  the  objectives  of  the  CECC.  Two  requirements  were  established  by  the  expert 
panel  at  the  ESG:  the  measures  must  use  existing  data  sources  and  the  data  gathering 
process  must  be  simple  enough  to  be  consistently  repeatable  by  the  CECC.  Furthermore, 
the  evaluation  measures  were  developed  according  to  the  desirability  criteria  explained  in 
Chapter  II. 

Natural  criteria,  or  criteria  for  which  there  is  a  direct,  quantitative  measurement,  is 
the  most  desirable.  Constructed  criteria,  which  indirectly  evaluates  the  degree  to  which 
an  alternative  contributes  to  the  associated  objective  based  on  qualitative  data  obtained 
from  a  subject  matter  expert,  is  second  in  the  hierarchy  of  desirable  evaluation  criteria. 
Proxy  criteria,  the  least  desirable,  uses  available  quantitative  data  that  indirectly  measures 
the  alternative’s  contribution  toward  the  objective  in  question  when  a  direct  means  of 
measurement  is  not  possible  (Keeney,  1992). 

Using  this  guidance,  the  ESG  panel  of  subject  matter  experts  developed  the 
evaluation  criteria  shown  in  Table  3.  A  brief  description  of  each  evaluation  measure  is 
listed  with  its  associated  objective  name  along  with  the  name  of  the  measure  type. 
Detailed  descriptions  of  the  data  used  to  score  alternatives  according  to  these  criteria  are 
listed  in  the  section  for  process  step  7. 
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Table  3.  Model  Evaluation  Measures 


Objective 

Measure 

Type 

1 

Reduce  first  costs 

Average  total  spent  per  FY  on  the  service  or 
commodity 

Natural 

2 

Reduce  O&M 

costs 

Average  total  spent  per  FY  on  O&M  for  the 
commodity 

Natural 

3 

Reduce  final  costs 

SME  evaluation  of  the  range  of  potential  final 
cost  savings 

Constructed 

4 

Improve  quality 

SME  evaluation  of  current  quality  issues  in  the 
commodity  or  service  area 

Constructed 

5 

Reduce  processing 
time 

Average  number  of  contracts  executed  per  FY 

Proxy 

6 

Leverage  new 
technology 

SME  evaluation  of  available  new  technology 

Constructed 

7 

Streamline 

regulations 

SME  evaluation  of  currency  and  consolidation 
potential  of  existing  regulations,  standards,  and 
guidance  pertaining  to  the  commodity  or  service 
area 

Constructed 

8 

Minimize  small 
business  risk 

Percentage  of  total  contracts  per  FY  awarded  to 
small  businesses 

Proxy 

9 

Reduce  utility 
usage 

SME  evaluation  of  percentage  of  possible  utility 
reductions 

Constructed 

10 

Improve 

standardization 

SME  evaluation  of  the  number  of  current  distinct 
solutions  for  the  requirement 

Constructed 

11 

Introduce  COLs 

SME  evaluation  of  percentage  of  demand 
reduced  by  the  implementation  of  applicable 

COLs 

Constructed 

Based  on  the  input  of  the  ESG  panel  of  experts,  natural  evaluation  measures  were 
only  possible  for  objectives  1  and  2  since  actual  cost  data  is  available  for  all  services  and 
commodities  purchased  by  the  CE  community.  Objective  5,  reduce  processing  time,  was 
assigned  a  proxy  evaluation  measure  since  it  is  not  feasible  to  measure  the  total  time 
personnel  across  the  entire  Air  Force  spend  working  on  contracts  related  to  a  specific 
opportunity.  Instead,  the  total  number  of  contracts  executed  per  FY  can  serve  as  a  proxy 
measure,  as  the  total  time  spent  across  the  Air  Force  is  closely  related  to  the  number  of 
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contracts  in  development.  Similarly,  objective  8,  minimize  small  business  risk,  does  not 
have  a  direct,  objective  means  of  measurement.  A  proxy  evaluation  measure  was 
assigned  that  uses  the  total  percentage  of  contracts  that  are  awarded  to  small  businesses. 
This  measure  can  approximate  the  level  of  adverse  impact  on  small  business  objectives 
possible  by  pursuing  a  strategic  sourcing  solution.  The  rest  of  the  objectives  were 
assigned  constructed  evaluation  measures  that  focus  on  the  professional  opinions  of  the 
opportunity’s  Subject  Matter  Expert  (SME)  at  AFCEC.  These  measures  utilize  an 
interview  process  with  the  SME  asking  specifically  defined  questions  developed  to  make 
the  evaluation  as  objective  as  possible.  The  questionnaire  used  during  the  interview 
process  for  each  of  the  SME  interviews  is  included  in  Appendix  B. 

Step  4:  Develop  Alternatives 

Alternatives  were  developed  by  first  examining  existing  documents  pertaining  to 
the  establishment  of  the  CECC.  During  the  initial  phases  of  planning  for  the  CECC’s 
first  contract  targets,  a  Commodity  Management  Plan  (CMP)  was  developed  to  both 
provide  internal  direction  to  CECC  personnel  as  well  as  to  forecast  potential  efficiencies 
for  budgeting  purposes.  During  this  process,  the  CECC  developed  a  prioritized  list  of 
strategic  sourcing  opportunities  according  to  the  original  spend  analysis  method 
discussed  in  Chapter  II.  The  prioritized  opportunities  listed  in  the  CMP  were  therefore 
selected  as  alternatives  to  facilitate  comparison  of  the  original  opportunity  assessment 
model  with  the  model  developed  during  this  research  project.  Table  4  lists  the  prioritized 
strategic  sourcing  opportunities  identified  in  the  CECC  CMP. 
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Table  4.  CMP -prioritized  Strategic  Sourcing  Opportunities  (Civil  Engineering 

Commodity  Council,  2011) 


Opportunity  Name 

Priority 

Taxiway  Lighting 

1 

Heating  Ventilation  and  Air  Conditioning  (HVAC)  Systems,  Chillers  and 
Boilers 

2 

Fire  Protection  and  Suppression 

3 

Rubber  Removal  and  Airfield  Restriping 

4 

Services  (including  Elevator  Maintenance,  Hood/Duct  Cleaning,  and  Dorm 
Appliance  Leasing/Maintenance 

5 

Energy  Monitoring  and  Control  Systems 

6 

Automatic  Gates  and  Doors 

7 

Paint  and  Protective  Coatings 

8 

Flooring 

9 

Roofing 

10 

Generators 

11 

Paved  Surface  Striping,  Painting,  and  Marking 

12 

Since  the  CMP  was  approved  in  201 1,  the  ESG  had  conducted  research  on  several 
of  the  listed  items  that  indicated  they  were  not  opportune  candidates  for  strategic 
sourcing.  Based  on  this  research  and  the  experience  of  the  ESG  engineers,  the  list  of 
CMP  commodities  and  services  to  be  evaluated  was  reduced  to  six.  In  addition,  the 
service  contract  category  originally  listed  as  fifth  on  the  CMP  priority  list  was  limited  to 
elevator  maintenance  only.  This  provided  a  sufficient  number  of  alternatives  to  compare 
with  the  CMP  priority  model  while  limiting  the  data  gathering  burden  to  a  manageable 
level. 

The  ESG  engineers  also  recommended  adding  three  alternatives  they  were  aware 
of  that  were  not  evaluated  and  prioritized  in  the  CMP  for  comparison  purposes.  Fire 
Personal  Protective  Equipment  (PPE),  HVAC  retrocommissioning,  and  water  leak 
detection  were  already  being  evaluated  in  an  opportunity  assessment  process.  The 
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priorities  of  these  three  alternatives  in  the  new  opportunity  assessment  model  were 


compared  with  the  priorities  of  the  services  and  commodities  already  identified  in  the 
CMP  in  order  to  provide  insight  into  their  relative  merit  in  advancing  the  goals  of  the 
strategic  sourcing  program. 

Finally,  during  the  subject  matter  expert  interview  process  described  in  step 
seven,  experts  were  asked  to  identify  additional  alternatives  for  consideration.  Although 
this  research  project  did  not  include  the  evaluation  of  these  alternatives,  they  are  provided 
for  future  evaluation  in  Chapter  IV.  Table  5  lists  the  outcome  of  the  CMP  review  and  the 
recommendations  made  by  the  ESG  staff.  These  nine  alternatives  underwent  evaluation 
according  to  the  model  developed  in  this  research  project.  Detailed  descriptions  of  each 
alternative  are  provided  in  Chapter  IV. 

Table  5.  List  of  Identified  Alternatives 


Opportunity  Name 

Source 

Elevator  Maintenance 

Commodity  Management  Plan 

Fire  Personal  Protective  Equipment 

ESG  Recommendation 

HVAC  Retrocommissioning 

ESG  Recommendation 

HVAC  Systems,  Chillers  and  Boilers 

Commodity  Management  Plan 

Roofing 

Commodity  Management  Plan 

Rubber  Removal  and  Airfield  Restriping 

Commodity  Management  Plan 

Generators 

Commodity  Management  Plan 

Taxiway  Lighting 

Commodity  Management  Plan 

Water  Leak  Detection 

ESG  Recommendation 

Step  5:  Create  Value  Functions 

The  correlation  between  an  alternative’s  raw  data  point  for  each  evaluation 
measure  and  its  associated  value  score  is  made  by  a  Single  Dimensional  Value  Function 
(SDVF).  SDVFs  can  be  either  linear,  curvilinear,  or  discrete,  but  must  either  increase  or 
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decrease  in  value  monotonically.  That  is,  the  slope  must  always  be  positive  or  negative 
over  the  entire  range  of  possible  values.  In  general,  indirect  evaluation  measures  result  in 
discrete  SDVFs,  while  direct  evaluation  measures  use  linear  SDVFs. 

The  first  step  to  developing  the  SDVFs  was  to  define  the  ranges  of  data  associated 
with  each  individual  evaluation  measure.  This  necessitated  that  the  SDVF  development 
process  occurred  after  the  data  had  been  gathered  for  the  alternatives  selected  in  the 
previous  step.  While  the  collected  data  is  detailed  in  Chapter  IV,  Table  6  below  displays 
the  data  ranges  for  the  evaluated  alternatives.  It  is  also  important  to  note  that  if  new 
alternatives  are  evaluated  with  this  opportunity  assessment  model,  the  data  must  be 
checked  to  ensure  it  falls  within  the  upper  and  lower  bounds  shown  in  the  table.  If  not,  it 
will  be  necessary  to  alter  the  SDVFs  to  account  for  the  new  range. 
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Table  6.  Data  Ranges  for  Evaluation  Measures 


Objective 

Measure 

Lower 

Bound 

Upper 

Bound 

1 

Reduce  first 

costs 

Average  total  spent  per  FY  on  the  service  or 
commodity 

$0 

S92.3M 

2 

Reduce  O&M 

costs 

Average  total  spent  per  FY  on  O&M  for  the 
commodity 

$0 

S52.37M 

3 

Reduce  final 

costs 

SME  evaluation  of  the  range  of  potential 
final  cost  savings 

0-2% 

>25% 

4 

Improve 

quality 

SME  evaluation  of  current  quality  issues  in 
the  commodity  or  service  area 

Never 

Constant 

5 

Reduce 

processing 

time 

Average  number  of  contracts  executed  per 

FY 

0 

307 

6 

Feverage  new 
technology 

SME  evaluation  of  available  new 
technology 

No 

Yes 

7 

Streamline 

regulations 

SME  evaluation  of  currency  and 
consolidation  potential  of  existing 
regulations,  standards,  and  guidance 
pertaining  to  the  commodity  or  service  area 

No 

Yes 

8 

Minimize 
small  business 
risk 

Percentage  of  total  contracts  per  FY 
awarded  to  small  businesses 

0 

100 

9 

Reduce  utility 
usage 

SME  evaluation  of  percentage  of  possible 
utility  reductions 

0-2% 

>25% 

10 

Improve 

standardization 

SME  evaluation  of  the  number  of  current 
distinct  solutions  for  the  requirement 

1 

>11 

11 

Introduce 

COFs 

SME  evaluation  of  percentage  of  demand 
reduced  by  the  implementation  of  applicable 
COFs 

0-2% 

>25% 

The  next  step  in  developing  SDVFs  is  to  determine  the  relationship  between  the 
possible  data  scores  within  the  ranges  that  were  identified  and  the  desired  value  score. 
Since  value  scores  for  each  evaluation  measure  can  range  from  0  to  1,  the  decision-maker 
must  decide  whether  the  minimum  data  value  will  achieve  a  zero  value  score  or  vice- 
versa.  This  will  determine  whether  or  not  the  SDVF  will  be  increasing  or  decreasing. 
Finally,  the  decision-maker  must  determine  the  relationship  between  the  data  and  the 
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value  score  within  the  range  of  possible  data  values.  For  linear  SDVFs,  this  is  achieved 
by  use  of  a  mathematical  relationship  between  the  data  point  and  the  value  score.  For 
discrete  SDVFs,  each  category  must  be  individually  assigned  a  corresponding  value 
score.  The  following  sections  explain  in  detail  the  development  process  for  each  of  the 
eleven  SDVFs  developed  for  this  opportunity  assessment  model. 

The  SDVF  for  reduce  first  cost  shown  graphically  in  Figure  9  is  a  linear, 
monotonically  increasing  function  ranging  between  $0  and  $92.3  Million.  Cost  data  for 
alternatives  that  contained  multiple  fiscal  years  was  averaged  across  the  years  collected. 
The  upper  bound  was  determined  based  on  the  maximum  calculated  average  annual  first 
cost  value  for  the  alternatives  considered.  This  function  yields  a  value  of  1  for  the  upper 
bound  of  $92.3  Million,  and  decreases  linearly  to  zero  as  the  annual  first  cost  approaches 
zero.  The  first  cost  data  was  obtained  for  each  alternative  by  the  ESG  from  the 
Commander’s  Resource  Information  System  (CRIS).  The  SDVF  is  defined  by  equation 
3.1,  where  v  is  the  objective  value  score,  x  is  the  alternative’s  average  annual  first  cost, 
and  x max  is  the  maximum  average  annual  first  cost  value  for  all  the  alternatives 
considered. 


v(x )  — - 

Xmax 


(3.1) 
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Figure  9.  SDVF  for  Reduce  First  Costs 


The  SDVF  for  reduce  O&M  costs  shown  graphically  in  Figure  10  is  a  linear, 
monotonically  increasing  function  ranging  between  $0  and  $52.37  Million.  Cost  data  for 
alternatives  that  contained  multiple  fiscal  years  was  averaged  across  the  years  collected. 
The  upper  bound  was  determined  based  on  the  maximum  calculated  average  annual 
O&M  spend  for  the  alternatives  considered.  This  function  yields  a  value  of  1  for  the 
maximum  average  annual  O&M  cost,  and  decreases  linearly  to  zero  as  average  first  cost 
approaches  zero.  The  O&M  cost  data  was  obtained  for  each  alternative  by  the  ESG  from 
CRIS.  The  SDVF  is  defined  by  Equation  3.2,  where  v  is  the  objective  value  score,  x  is 
the  alternative’s  average  annual  O&M  cost,  and.v„(ar  is  the  maximum  average  annual 
O&M  cost  for  all  the  alternatives  considered. 

v(x)  =  (3.2) 

xmax 
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Figure  10.  SDVF  for  Reduce  O&M  Costs 


The  SDVF  for  reduce  final  costs  is  a  categorical,  monotonically  increasing 
function  based  on  the  evaluation  of  potential  for  a  final  cost  reduction  by  the  subject 
matter  expert  at  AFCEC.  The  categories  were  separated  into  five  percent  ranges,  with  a 
value  of  0.25  being  assigned  to  each  subsequent  category.  Figure  1 1  displays  the 
percentage  range  for  each  category  and  its  corresponding  value  score. 
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0-2%  3-5%  5-10%  10-25%  >25% 

Estimated  Percent  Reduction  in  Final  Costs 


Figure  11.  SDVF  for  Reduce  Final  Costs 


The  SDVF  for  improve  quality  is  a  categorical,  monotonically  increasing  function 
based  on  the  evaluation  of  the  relative  quantity  of  quality  issues  that  currently  arise  with 
non-strategically  sourced  contracts  by  the  subject  matter  expert  at  AFCEC.  There  are 
five  categories,  with  a  value  of  0.25  being  assigned  to  each  subsequent  category.  Figure 
12  displays  the  response  for  each  category  and  its  corresponding  value  score. 
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Never  Rarely  Occasionally  Often  Constantly 

Evaluation  of  Current  Quality  Issues 


Figure  12.  SDVF  for  Improve  Quality 

The  SDVF  for  reduce  processing  time  shown  in  Figure  13  is  a  linear, 
monotonically  increasing  function  ranging  between  0  and  307.  The  maximum  was 
determined  based  on  the  average  number  of  contracts  executed  per  Fiscal  Year  (FY)  for 
the  alternatives  considered.  This  function  yields  a  value  of  1  for  the  maximum  number  of 
contracts  executed,  and  decreases  linearly  to  zero  as  the  number  of  contracts  executed 
approaches  zero.  Average  number  of  contracts  executed  per  FY  was  calculated  for  each 
alternative  by  averaging  the  total  number  of  contracts  collected  by  the  ESG  from  the 
CRIS  over  the  number  of  fiscal  years  collected.  Equation  3.3  defines  the  SDVF  where  v 
is  the  objective  value  score,  x  is  the  alternative’s  average  number  of  executed  contracts 
per  FY,  and  xmax  is  the  maximum  number  of  executed  contracts  per  FY  for  all  the 
alternatives  considered. 

v(x)  =  (3.3) 

xmax 
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Figure  13.  SDVF  for  Reduce  Processing  Time 

The  SDVF  for  leverage  new  technology  is  a  categorical,  binary  function  with 
possible  values  of  “yes”  and  “no”.  This  assessment  is  based  on  the  evaluation  of  the 
availability  of  a  new  technology  pertinent  to  a  strategic  sourcing  opportunity  by  the 
subject  matter  expert  at  AFCEC. 
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Figure  14.  SDVF  for  Leverage  New  Technology 
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The  SDVF  for  streamline  regulations  is  a  categorical,  binary  function  with 
possible  values  of  “yes”  and  “no”.  This  assessment  is  based  on  the  evaluation  of  the  need 
for  updating  or  consolidating  guidance,  standards,  or  regulations  pertaining  to  a  strategic 
sourcing  opportunity  by  the  subject  matter  expert  at  AFCEC. 


Do  regulations  require  updating  or  consolidating? 


Figure  15.  SDVF  for  Streamline  Regulations 


The  SDVF  for  minimize  small  business  risk  is  a  monotonically  decreasing 
function  that  assigns  a  value  score  to  alternatives  based  on  the  percentage  of  total 
contracts  that  are  awarded  to  small  business.  This  relationship  was  determined  to  assign 
a  higher  value  score  for  opportunities  that  have  a  higher  percentage  of  small  business 
involvement.  A  non-linear  function  was  used  to  emphasize  the  negative  desirability  for 
opportunities  that  were  heavily  or  exclusively  awarded  to  small  businesses.  The 
percentage  was  calculated  by  dividing  the  total  number  of  contracts  awarded  to  small 
businesses  by  the  total  number  of  contracts  awarded  from  contract  data  gathered  from 
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CBIS.  The  relationship  is  defined  by  Equation  3.1  and  displayed  graphically  in  Figure 


16.  In  the  equation,  v  represents  the  value  score  for  this  objective  and  x  represents  the 
percentage  of  contracts  awarded  to  small  businesses. 

=  1  -  (£)  <3-‘> 


1.2 


Figure  16.  SDVF  for  Minimize  Small  Business  Risk 

The  SDVF  for  reduce  utility  usage  is  a  categorical,  monotonically  increasing 
function  based  on  the  evaluation  of  the  potential  percent  reduction  in  utility  usage  by  the 
subject  matter  expert  at  AFCEC.  There  are  five  categories,  with  a  value  of  0.25  being 
assigned  to  each  subsequent  category.  Figure  17  displays  the  percentage  breakdown  for 
each  category  and  its  corresponding  value  score. 
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Estimated  Percent  Reduction  in  Utility  Usage 


Figure  17.  SDVF  for  Reduce  Utility  Usage 

The  SDVF  for  improve  standardization  is  a  categorical,  monotonically  increasing 
function  based  on  the  evaluation  of  the  total  number  of  distinct  solutions  currently  in  use 
for  the  opportunity  being  assessed  by  the  subject  matter  expert  at  AFCEC.  There  are  four 
categories,  with  a  value  of  0.333  being  assigned  to  each  subsequent  category.  Figure  18 
displays  the  range  of  solution  types  for  each  category  and  its  corresponding  value  score. 


Figure  18.  SDVF  for  Improve  Standardization 
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The  SDVF  for  introduce  COLs  is  a  categorical,  monotonically  increasing  function 
based  on  the  evaluation  of  the  potential  percent  reduction  in  service  demand  due  to  the 
proposed  COL  by  the  subject  matter  expert  at  AFCEC.  There  are  five  categories,  with  a 
value  of  0.25  being  assigned  to  each  subsequent  category.  Figure  17  displays  the 
percentage  breakdown  for  each  category  and  its  corresponding  value  score. 


0-2%  3-5%  5-10%  10-25%  >25% 

Estimated  Percent  Reduction  in  Demand 


Figure  19.  SDVF  for  Introduce  COLs 


Step  6:  Develop  Weights 

To  assign  a  relative  level  of  importance  to  each  objective,  weight  factors  were 
developed  for  each  of  the  objectives  contained  in  the  objective  hierarchy.  Weights  were 
assigned  in  tier  groups,  with  tier  one  objectives  being  weighted  first,  followed  by  each 
sub-objective  family  group  in  tier  two.  The  weight  of  the  parent  objective  is  then  applied 
to  each  of  its  sub-objectives,  giving  rise  to  the  concept  of  global  and  local  weights.  Local 
weights  are  the  factors  directly  assigned  to  each  sub-objective  in  the  preliminary 
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weighting  process,  and  global  weights  are  the  overall  weight  factor  assigned  to  an 
objective.  Global  weights  are  simply  the  product  of  the  multiplication  of  the  local  weight 
by  the  parent  objective’s  weight.  The  following  paragraphs  detail  the  specific 
methodology  used  in  this  research  project  to  assign  weight  factors  to  each  objective  in  the 
objective  hierarchy. 

Similar  to  the  objective  hierarchy  creation  process,  two  groups  of  experts  were 
used  to  assign  weight  factors.  Engineers  from  the  ESG  were  first  asked  to  assign  weights 
to  the  objective  hierarchy  in  a  top-down  manner.  That  is,  the  first  tier  objectives  were 
weighted  first,  followed  by  each  set  of  sub-objectives  on  the  second  tier  subordinate  to  a 
first  tier  objective.  The  “100  coin”  method  explained  in  Chapter  II  was  used,  where 
panel  members  were  asked  to  conceptually  divide  a  group  of  100  coins  among  the 
objective  according  to  the  relative  importance  of  each.  Individual  opinions  were 
identified  to  the  group,  and  the  group  converged  on  a  solution.  Each  score  from  0  to  100 
corresponded  to  the  percent  weight  assigned  to  the  objective,  ranging  from  0  to  1.  Once 
the  ESG  panel  completed  their  weight  factors,  the  list  of  weights  was  submitted  to  the 
panel  of  experts  at  AFCEC  for  review.  The  AFCEC  revisions  were  submitted  again  to 
the  ESG  panel,  who  adjusted  their  weighting  responses  and  produced  the  final  list  of 
weights. 

Once  the  entire  hierarchy  was  weighted,  the  weight  system  was  validated  by  ESG 
leadership  by  examining  the  weight  factors  from  the  bottom  up.  That  is,  ESG  leadership 
looked  at  the  outcome  of  the  global  weight  factors  of  each  sub-objective  relative  to  each 
other  to  determine  whether  the  system  was  consistent  with  the  priorities  of  the  senior 
CECC  decision-makers.  Figure  20  and  Figure  21  graphically  display  the  relative  global 
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weights  of  each  of  the  objectives  contained  within  the  objective  hierarchy.  Table  7 
contains  the  specific  numerical  weights  assigned  to  each  objective  based  on  the  method 
explained  above. 


■  Create  Rate  Savings 

■  Improve  Processes 

■  Manage  Demand 


Figure  20.  Tier  1  Objective  Weights 


■  Reduce  First  Costs 

■  Reduce  O&M  Costs 

■  Reduce  Final  Costs 

■  Improve  Quality 

■  Reduce  Processing  Time 

■  Leverage  New  Technology 

■  Streamline  Regulations 

■  Minimize  Small  Business  Risk 
Reduce  Utility  Usage 

■  Improve  Standardization 
Establish  COLs 


Figure  21.  Tier  2  Objective  Weights 
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Table  7.  Local  and  Global  Weight  Factors  for  Model  Objectives 


Objective  Name 

Local  weight 

Global  Weight 

Create  Rate  Savings 

0.3640 

Reduce  first  costs 

0.40 

0.1456 

Reduce  O&M  costs 

0.30 

0.1092 

Reduce  final  costs 

0.10 

0.0364 

Improve  quality 

0.20 

0.0728 

Improve  processes 

0.3640 

Reduce  touch  time 

0.25 

0.0910 

Leverage  new  technology 

0.20 

0.0728 

Streamline  regulations 

0.25 

0.0910 

Minimize  small  business  risk 

0.30 

0.1092 

Manage  Demand 

0.2720 

Reduce  utility  expenses 

0.40 

0.1088 

Establish  standard  solutions 

0.30 

0.0816 

Establish  COLs 

0.30 

0.0816 

Step  7:  Determine  Overall  Values  for  Alternatives 

Steps  1-3  and  5  of  the  VFT  process  provide  the  various  components  that  comprise 
the  overall  decision  model  for  the  problem.  These  components  can  be  combined  into  a 
single  mathematical  equation  known  as  the  value  equation.  The  value  equation 
determines  overall  value  scores  for  each  alternative.  This  score  can  be  used  to  compare 
the  relative  alignment  of  each  alternative  with  the  objectives  of  the  decision  maker.  The 
value  equation  for  each  objective,  shown  below  in  equation  3.2,  consists  of  two  main 
parts:  the  objective  weight  and  the  SDVF  for  the  alternative  that  is  being  evaluated.  Each 
of  the  n  objectives  has  its  own  weight  factor,  SDVF,  and  data  input.  The  sum  of  the 
value  scores  for  each  of  the  objectives  is  the  overall  value  score  for  the  alternative.  In  the 
equation,  v(x)  represents  the  value  score,  w,-  represents  the  global  weight  factor,  and  Vi(x) 
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represents  the  SDVF  for  the  ith  objective.  Table  8  provides  a  list  of  values  for  the 


variables  and  the  SDVF  definitions. 

v(x)  =  Z?=1WiVi(Xi)  (3.2) 


Table  8.  Value  Equation 


i 

Objective  Name 

Weight  Factor  (  w, ) 

SDVF  ( Vi(Xi)  ) 

1 

Reduce  first  costs 

0.1456 

Xi 

Ul(Xi)  = 

X1  max 

2 

Reduce  O&M  costs 

0.1092 

x2 

V2&2)  = 

x2  max 

3 

Reduce  final  costs 

0.0364 

Categorical  (0,  0.25,  0.5,  0.75,  1) 

4 

Improve  quality 

0.0728 

Categorical  (0,  0.25,  0.5,  0.75,  1) 

5 

Reduce  touch  time 

0.0910 

Xrr 

Vs(*5)  = 

x5  max 

6 

Leverage  new  technology 

0.0728 

Categorical  (0,  1) 

7 

Streamline  regulations 

0.0910 

Categorical  (0,  1) 

8 

Minimize  small  business 
risk 

0.1092 

VsM-l  (100) 

9 

Reduce  utility  expenses 

0.1088 

Categorical  (0,  0.25,  0.5,  0.75,  1) 

10 

Establish  standard 
solutions 

0.0816 

Categorical  (0,  0.333,  0.666,  1) 

11 

Establish  COLs 

0.0816 

Categorical  (0,  0.25,  0.5,  0.75,  1) 

Once  the  value  function  was  created,  data  for  each  of  the  alternatives  was  needed 
in  order  to  calculate  the  associated  value  scores.  As  explained  previously,  several  of  the 
objectives  were  able  to  be  measured  directly  with  natural  or  proxy  evaluation  measures, 
while  others  were  measured  indirectly  with  constructed  evaluation  measures.  In  general, 
all  natural  and  proxy  evaluation  measures  used  data  collected  from  CRIS  by  ESG 
engineers,  while  all  constructed  attributes  used  data  obtained  through  interviews  with 
subject  matter  experts  from  AFCEC. 

Objectives  1,  2,  5,  and  8  were  measured  using  data  obtained  from  CRIS.  These 
reports  were  generated  by  ESG  personnel  during  their  opportunity  assessment  phase 
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evaluations  of  various  alternatives  under  the  current  opportunity  assessment  system. 

Using  this  existing  data  allowed  for  a  closer  comparison  between  the  two  opportunity 
assessment  models,  and  greatly  decreased  the  workload  on  the  ESG  engineers  related  to 
this  research  effort.  Because  the  data  sets  were  created  for  independent  projects,  there  is 
some  variation  in  the  time  ranges  and  the  specific  data  fields  included  in  each  database. 
The  data  was  typically  contained  within  a  Microsoft  Excel  spreadsheet,  although  some 
spreadsheets  had  been  imported  into  Microsoft  Access  database  files  to  enable  a  higher 
degree  of  interaction.  Most  databases  contained  data  for  fiscal  years  2010-2012.  The 
data  used  is  available  from  the  author  upon  request. 

The  remaining  objectives  were  measured  using  data  obtained  from  interviews 
with  the  subject  matter  experts  at  AFCEC  for  each  of  the  services  or  commodities  that 
were  evaluated.  Interviews  were  primarily  conducted  over  the  phone;  however,  several 
of  them  were  completed  via  a  written  questionnaire  sent  and  received  through  e-mail.  A 
list  of  questions  and  definitions  of  response  categories  is  included  in  Chapter  IV.  The  full 
questionnaire  is  included  in  Appendix  A. 

Once  the  data  were  collected  for  each  alternative,  it  was  recorded  on  a  Microsoft 
Excel  spreadsheet.  A  value  score  for  each  evaluation  measure  was  then  calculated.  The 
spreadsheet  was  used  to  sum  the  objective  value  scores  to  find  the  overall  value  score  for 
each  alternative.  Results  of  the  value  scores  for  each  alternative  along  with  the 
prioritized  list  of  opportunities  are  provided  in  Chapter  IV. 
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Step  8:  Select  Alternative 

The  result  of  the  alternative  evaluation  was  a  value  score  for  each  of  the 
alternatives  considered.  Value  scores  can  range  from  0  to  1,  with  1  being  the  most 
desirable.  These  value  scores  provide  an  objective  basis  for  the  comparison  of  the 
alternatives  considered.  The  complete  list  of  value  scores  for  the  alternatives  is  included 
in  Chapter  IV.  In  addition  to  simply  producing  the  raw  value  scores  for  the  alternatives 
selected,  a  sensitivity  analysis  was  conducted  in  order  to  evaluate  how  variations  in 
weight  factors  influence  the  outcome  of  the  model. 

The  sensitivity  analysis  was  conducted  on  all  the  evaluated  alternatives  as  well  as 
two  dummy  alternatives  that  were  assigned  randomly-generated  data.  These  dummy 
alternatives  serve  to  allow  for  insight  into  how  other  previously  unconsidered  alternatives 
would  respond  to  changes  in  the  weighting  factors.  The  sensitivity  analysis  consisted  of 
varying  the  weight  of  each  factor,  and  observing  the  effect  on  the  resulting  value  scores 
for  each  alternative.  Results  from  this  process,  including  figures  and  tables,  are  included 
in  Chapter  Four 

Chapter  Summary 

This  chapter  described  the  methods  used  in  this  research  project  to  develop  the 
decision  support  model,  select  alternatives,  gather  the  necessary  data,  and  evaluate  the 
final  outcome  of  the  decision  analysis  process.  All  eight  of  the  VFT  process  steps  were 
described  in  detail.  The  next  chapter  provides  a  comprehensive  review  and  analysis  of 
the  results  of  the  research  project  and  an  analysis  of  the  data  and  the  implementation  of 
the  decision  model. 
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IV.  Results  and  Analysis 


The  purpose  of  this  chapter  is  to  provide  a  detailed  summary  of  the  results  of  this 
research.  The  objective  of  the  research  was  to  develop  a  decision  support  model  to  assist 
the  Civil  Engineer  Commodity  Council  (CECC)  with  opportunity  assessments.  The 
decision  model  takes  the  form  of  a  value  equation,  with  an  associated  value  hierarchy 
tying  the  model  to  the  strategic  objectives  of  the  CECC.  In  addition  to  the  creation  of  the 
decision  model,  the  results  of  the  Commodity  Management  Plan  (CMP)  opportunity 
assessment  were  compared  with  results  from  the  newly  developed  model  to  demonstrate 
the  model  characteristics.  The  results  of  the  comparison  study  are  presented  along  with  a 
sensitivity  analysis  of  the  weighting  factors  assigned  to  each  of  the  evaluation  criteria 
used  in  the  model. 

The  Opportunity  Assessment  Decision  Model 

To  understand  the  value  equation  developed  in  this  research,  the  value  hierarchy 
explained  in  Chapter  III  is  re-presented  in  its  entirety  in  Figure  22  along  with  the  value 
weights  and  associated  evaluation  factors.  Both  the  local  and  global  weights  are  provided 
for  each  objective.  The  value  hierarchy  consists  of  a  fundamental  objective  with  two 
tiers  of  subordinate  objectives.  Each  tier-two  objective  was  assigned  an  evaluation 
measure  to  determine  an  alternative’s  potential  to  further  the  accomplishment  of  the 
objective.  Each  evaluation  measure  corresponds  to  an  ith  variable  set  in  the  value 
equation  described  in  Chapter  III  and  used  to  determine  the  overall  score  for  each  of  the 
alternatives  that  was  evaluated. 
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Global 

Weight 

0.1456 

0.1092 

0.0364 

0.0728 

0.0910 

0.0728 

0.0910 

0.1092 

0.1088 

0.0816 

0.0816 


Figure  22.  The  CECC  Opportunity  Assessment  Decision  Model 
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Analysis  of  Alternatives 


Alternatives  were  evaluated  using  the  value  equation  described  in  Chapter  III. 
Data  entered  into  the  model  for  each  alternative  were  collected  from  either  spend  and 
contract  data  collected  from  the  ESG  or  from  interviews  conducted  with  subject  matter 
experts  from  AFCEC  as  described  in  Chapter  III.  Data  was  recorded  using  a  Microsoft 
Excel  spreadsheet  programmed  to  calculate  the  value  scores  for  each  alternative 
according  to  the  value  equation.  The  final  prioritized  list  of  the  nine  evaluated 
alternatives  is  presented  in  Table  9. 


Table  9.  Results  of  the  Alternative  Assessment 


Rank 

Alternative 

Value 

Score 

1 

HVAC  Equipment 

0.5830 

2 

Roofing 

0.4853 

3 

Fire  Protection  PPE 

0.3361 

4 

Generators 

0.2874 

5 

Water  Leak  Detection 

0.2731 

6 

Grounds  Maintenance 

0.2751 

7 

Elevator  Maintenance 

0.2866 

8 

Taxiway  Lighting 

0.2480 

9 

Runway  Rubber  Removal  and  Restriping 

0.1594 

Figure  23  provides  a  visual  representation  of  the  individual  objective  scores 
associated  with  each  alternative.  This  graph  can  be  used  to  see  the  relative  impact  of 
each  objective  on  the  overall  score  for  the  alternatives.  General  trends  can  also  be 
observed  regarding  the  relative  effect  each  objective  had  on  the  outcome  of  the 
alternative  analysis.  For  example,  the  single  largest  contributor  to  the  value  scores  of  the 
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alternatives  was  the  streamline  regulations  objective,  even  though  it  had  the  fifth  highest 
weighting  factor.  This  can  be  attributed  to  the  fact  that  the  objective  used  a  binary 
categorical  Single  Dimensional  Value  Function  (SDVF)  for  which  most  of  the 


alternatives  achieved  a  maximum  score. 


COLs 

■  Standardization 
Utility  Reduction 

■  Small  Business  Risk 

■  Regulations 

■  New  Technology 

■  Process  Time 

■  Quality 

■  Final  Cost 

■  O&M  Cost 

■  First  Cost 


Figure  23.  Value  Score  Breakdown  for  All  Alternatives 

As  the  data  in  Figure  23  indicates,  a  high  score  from  any  individual  objective  did 
not  necessarily  guarantee  a  high  ranking  for  the  alternative.  On  the  contrary,  no 
individual  objective  had  an  overpowering  influence  on  the  overall  priority  rankings  of  the 


58 


alternatives.  This  fact  indicates  that  the  model  was  well  balanced  between  all  of  the 
objectives,  and  did  not  rely  too  heavily  on  any  particular  objective. 

Another  important  consideration  of  the  analysis  of  this  research  is  the  comparison 
between  the  results  of  the  model  used  in  this  research  with  the  results  obtained  using  the 
existing  opportunity  assessment  method  captured  in  the  CECC  CMP.  As  stated  in 
Chapter  III,  to  make  a  valid  comparison  between  the  models,  the  same  data  was  used  to 
evaluate  the  alternatives  in  both  methods  to  the  maximum  extent  allowable  by  the  model. 
However,  since  the  new  model  developed  in  this  research  effort  evaluated  a  wider  range 
of  alternative  factors  based  on  the  developed  objective  hierarchy,  most  of  the  data  used  in 
the  new  model  was  not  considered  during  the  CMP  analysis  process.  The  objectives  that 
utilized  the  same  or  similar  data  were  reduce  first  cost,  reduce  O&M  cost,  reduce 
processing  time,  and  reduce  small  business  risk  objectives.  The  remaining  objectives 
included  in  the  new  model  used  objectives  outside  of  the  scope  of  the  original  model. 
Figure  24  displays  a  comparison  of  the  alternative  rankings  between  the  VFT  model  and 
the  CMP  opportunity  assessment  analysis.  Arrows  indicate  positional  changes  in  the 
rankings  between  the  two  methods.  In  general,  the  VFT  model  produced  drastically 
different  results  due  to  the  inclusion  of  the  additional  objectives  for  consideration. 
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CMP  Rankings 

Rank 

Alternative 

1 

Taxiway  Lighting 

2 

HVAC  Equipment 

3 

Runway  Rubber  Removal 
and  Re  striping 

4 

Elevator  Maintenance 

5 

Roofing 

6 

Generators 

VFT  Model  Rankings 

Rank 

Alternative 

1 

HVAC  Equipment 

2 

Roofing 

3 

*  Fire  Protection  PPE 

4 

Generators 

5 

*  Water  Leak  Detection 

6 

*  Grounds  Maintenance 

7 

Elevator  Maintenance 

8 

Taxiway  Lighting 

9 

Runway  Rubber  Removal 
and  Restriping 

*  Alternative  not  included  in  CMP 


Figure  24.  Comparison  of  Model  Alternative  Rankings 


The  difference  in  the  alternative  rankings  indicates  that  the  new  criteria  included 
in  the  VFT  alternative  evaluation  process  add  information  to  the  model  that  is 
independent  of  and  fundamentally  different  from  the  information  included  in  the  CMP 
model.  Because  inclusion  of  this  additional  information  is  justified  by  the  objectives  of 
the  strategic  sourcing  program  specified  in  the  objective  hierarchy,  the  new  model’s 
results  are  influenced  by  a  more  complete  picture  of  each  alternative’s  true  value  to  the 
strategic  sourcing  program.  This  indicates  the  new  model  is  successful  in  introducing 
new  criteria  in  the  evaluation  process,  thereby  strengthening  the  validity  of  the  model. 

Identification  of  New  Alternatives 

In  addition  to  the  alternatives  presented  in  the  CECC  Commodity  Management 
Plan  (CMP)  and  detailed  in  Chapter  III,  new  alternatives  were  developed  for  evaluation 
based  on  input  from  subject  matter  experts  at  the  Enterprise  Sourcing  Group  (ESG)  and 
the  Air  Force  Civil  Engineer  Center  (AFCEC).  Due  to  the  fact  that  the  data  required  to 
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complete  an  assessment  of  these  alternatives  had  not  already  been  collected  by  the  ESG, 


evaluation  of  these  alternatives  was  beyond  the  scope  of  this  research  effort.  The  list  of 
newly  identified  alternatives  and  their  source  is  provided  in  Table  10.  The  list  includes 
nine  new  alternatives  in  a  variety  of  different  functional  areas. 


Table  10.  New  Alternatives  Identified 


Alternative 

Source 

Base  Recycling  Services 

AFCEC 

Airfield  Pavement  Repair 

AFCEC 

Wastewater  Treatment  Privatization 

AFCEC 

Bridge  and  Dam  Inspections 

AFCEC/ESG 

Hazardous  Material  Response  Equipment 

AFCEC 

Fire  Response  Equipment  (non-PPE) 

AFCEC 

Fan  Coil  Units 

AFCEC 

Water  Source  Heat  Pumps 

AFCEC 

Automatic  Transfer  Switches 

AFCEC 

Sensitivity  Analysis 

As  part  of  the  research,  a  sensitivity  analysis  was  conducted  on  the  results  of  the 
alternative  evaluation.  Weight  factors  were  varied  for  each  objective  from  0  to  0.2,  and 
the  resulting  value  scores  for  each  alternative  were  tabulated  using  Microsoft  Excel.  0.2 
was  chosen  as  the  maximum  weight  used  in  the  analysis  because  no  significant  changes 
in  the  results  occurred  when  weight  factors  were  increased  to  greater  than  0.2.  In 
addition,  a  simulated  service  contract  alternative  and  a  simulated  commodity  alternative 
were  created  using  data  generated  by  the  Excel  random  number  generation  tool.  The 
data  was  created  by  causing  the  random  number  generation  tool  to  generate  an  integer 
between  the  minimum  and  maximum  values  of  the  data  found  in  the  alternatives  that 
were  analyzed.  This  random  number  was  then  used  to  calculate  an  overall  value  score  for 
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the  alternative  using  the  value  equation.  The  results  from  the  sensitivity  analysis  are 
presented  in  Figures  24-34  as  spider  diagrams. 

Each  spider  diagram  displays  the  value  scores  of  each  alternative  as  the  weight  of 
a  specific  objective  is  varied  from  0  to  0.2  as  a  line.  Because  value  scores  are  always 
positive,  the  lines  will  always  have  a  positive  slope.  Horizontal  lines  indicate  that  the 
alternative  received  a  zero  value  score  for  the  objective  being  analyzed  in  the  graph.  The 
vertical  line  in  each  figure  represents  the  original  weight  factor  assigned  in  the  model. 
The  priority  ranking  of  the  alternatives  can  be  determined  at  any  weight  by  observing  the 
relative  value  score  for  each  alternative  at  the  weight  factor  being  considered. 
Alternatives  at  the  top  will  rank  higher  than  those  below  it.  As  a  result,  when  lines  cross, 
a  change  in  the  priority  ranking  occurs  at  the  weight  where  the  intersection  is  located. 

The  first  objective,  reduce  first  cost,  had  a  global  weight  of  0.1456.  The 
sensitivity  analysis  indicates  that  five  alternatives  were  sensitive  to  changes  in  the 
objective  weight:  generators,  elevator  maintenance,  grounds  maintenance,  taxiway 
lighting  and  water  leak  detection.  While  the  top  four  and  bottom  two  alternatives  do  not 
vary  significantly  over  the  range  of  the  analysis,  the  ranking  of  the  other  five  alternatives 
do  vary  significantly.  The  sensitivity  analysis  results  are  presented  in  the  spider  diagram 
shown  in  Figure  25. 
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Figure  25.  Sensitivity  Analysis  for  Reduce  First  Cost 


The  second  objective,  reduce  O&M  costs,  had  an  initial  weight  factor  of  0.1092. 
Figure  26  displays  a  spider  diagram  of  the  sensitivity  analysis  for  the  alternatives 
considered  and  the  additional  simulated  alternatives.  The  analysis  indicates  that  this  is  a 
relatively  stable  objective  less  sensitive  to  changes  in  the  weighting  factor  for  the 
alternatives  considered.  The  only  difference  in  the  outcome  occurs  if  the  objective  is 
assigned  a  weight  factor  over  0.17.  This  is  due  mainly  to  the  fact  that  the  O&M  cost  data 
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for  the  alternative  was  not  available  for  inclusion  in  the  analysis,  or  that  O&M  costs  are 


not  applicable  to  the  alternative,  as  is  the  case  with  service  contracts. 
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Figure  26.  Sensitivity  Analysis  for  Reduce  O&M  Cost 


The  third  objective,  reduce  final  costs  had  an  initial  weight  factor  of  0.0364.  The 
sensitivity  analysis  indicated  no  significant  changes  in  results  for  weight  factors  below 
0.12.  Above  0.12,  roofing  and  LED  taxiway  lighting  scores  increase  enough  to  alter  the 
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rankings.  Horizontal  lines  indicate  the  alternative  received  a  score  of  0  for  this  objective 
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Figure  27.  Sensitivity  Analysis  for  Reduce  Final  Costs 


The  sensitivity  analysis  for  the  fourth  objective,  improve  quality,  indicated  that 
slight  changes  to  the  weight  factors  will  affect  the  model  results.  The  original  weight 
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factor  assigned  was  0.0728.  Five  changes  in  the  rank  order  of  the  alternatives  occur 
between  weight  factors  of  0.02  and  0.13  as  indicated  in  Figure  28. 


Figure  28.  Sensitivity  Analysis  for  Improve  Quality 

Figure  29  displays  the  sensitivity  analysis  for  the  objective  to  reduce  processing 
time.  The  weight  factor  for  this  objective  in  the  decision  model  is  0.091.  A  slight 
decrease  in  the  objective  weight  will  cause  elevator  maintenance  to  overtake  generators 
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on  the  alternative  rankings.  Other  less  significant  changes  occur  as  indicated  on  the 
spider  diagram. 


Figure  29.  Sensitivity  Analysis  for  Reduce  Processing  Time 

The  sixth  objective,  leverage  new  technology,  has  a  model  objective  weight  of 
0.0728.  Because  of  the  binary  nature  of  this  objective’s  SDVF,  alternatives  with  a 
positive  value  score  in  this  objective  will  have  dramatically  changing  scores  as  the 
objective  weight  is  manipulated  as  indicated  in  Figure  30.  The  elevator  maintenance 
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alternative  is  indicative  of  this  as  either  a  slight  decrease  or  a  slight  increase  in  the 
objective  weight  will  affect  the  alternative  rankings. 
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Figure  30.  Sensitivity  Analysis  for  Leverage  New  Technology 


Similar  to  the  previous  objective,  the  model  results  for  streamline  regulations  is 
greatly  affected  by  the  assigned  weight  factor  as  it  also  has  a  binary  SDVF.  The  model 
weight  factor  for  this  objective  is  0.091.  The  priority  rankings  of  elevator  maintenance, 
generators,  water  leak  detection,  grounds  maintenance,  and  taxiway  lighting  change  six 
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times  as  the  weight  is  varied  between  0.07  and  0.13.  Figure  31  displays  additional 
changes  in  the  alternative  rankings  as  the  weight  factor  is  varied. 
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Figure  31.  Sensitivity  Analysis  for  Streamline  Regulations 


The  eighth  objective,  minimize  small  business  risk,  was  the  most  stable  of  the 
objectives  to  variations  in  objective  weight.  While  this  objective  has  a  high  weight  factor 
of  0.1092  in  the  model,  the  exponentially  decreasing  SDVF  resulted  in  a  low  magnitude 
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for  all  the  alternatives  evaluated.  Figure  32  displays  the  full  result  of  the  sensitivity 
analysis.  This  is  partially  due  to  lack  of  contract  data  for  many  of  the  alternatives. 
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Figure  32.  Sensitivity  Analysis  for  Minimize  Small  Business  Risk 


The  objective  to  reduce  utility  expenses  has  a  high  model  weight  factor  of  0.1088. 
As  shown  in  Figure  33,  variations  in  the  weight  factor  significantly  affected  the  value 
scores  for  those  alternatives  that  exhibited  utility  reduction  potential.  The  alternative 
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most  sensitive  to  changes  in  the  objective  weight  was  water  leak  detection.  As  the 
objective  weight  is  varied  between  0.05  and  0.13,  four  changes  in  the  alternative  priority 
ranking  occur. 
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Figure  33.  Sensitivity  Analysis  for  Reduce  Utility  Expenses 


71 


The  tenth  objective,  establish  standard  solutions  is  the  most  sensitive  of  the 
objectives  to  changes  in  the  objective  weight.  This  is  due  to  the  widely  varying  scores 
assigned  to  each  of  the  alternatives.  Thirteen  rank  changes  are  identified  within  the  range 
of  weights  used  in  this  sensitivity  analysis.  The  model  weight  for  this  objective  is 
0.0816.  Results  of  the  analysis  are  displayed  in  Figure  34. 


Figure  34.  Sensitivity  Analysis  for  Establish  Standard  Solutions 
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The  final  objective,  establish  COLs,  has  a  model  weight  of  0.0816.  Because  this 
objective  is  only  applicable  to  service  contracts,  the  scores  of  most  alternatives  did  not 
vary  over  the  range  of  the  analysis.  Elevator  maintenance,  grounds  maintenance,  and  the 
simulated  service  contract  alternative  were  greatly  affected  by  the  weight  changes,  with 
seven  changes  occurring  in  the  alternative  rankings  as  the  objective  weight  was  varied. 
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Figure  35.  Sensitivity  Analysis  for  Establish  COLs 
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Overall,  the  sensitivity  analysis  indicates  that  the  results  of  the  analysis  are  highly 
sensitive  to  changes  to  the  weight  factors  assigned  to  each  objective.  This  fact 
underscores  the  need  for  the  decision-maker  to  fully  understand  the  effect  of  each  weight 
factor  on  the  overall  outcome  of  the  model,  and  to  ensure  that  an  appropriate  amount  of 
care  is  taken  when  objective  weight  factors  are  assigned. 

Conclusion 

This  chapter  presented  the  results  of  the  research,  including  both  the  VFT-based 
decision  model  for  the  CECC  opportunity  assessment,  and  the  analysis  of  selected 
alternatives.  In  addition,  the  results  of  the  alternative  analysis  were  compared  with  the 
analysis  results  of  the  decision  model  currently  in  use  by  the  CECC,  demonstrating  the 
significance  of  the  new  information  included  in  the  analysis  based  on  the  full  range  of 
strategic  sourcing  objectives  found  in  the  objective  hierarchy.  A  sensitivity  analysis  was 
presented  detailing  the  difference  in  results  of  the  model  as  the  weighting  factors  for  each 
objective  were  varied  within  a  specified  range,  and  highlighted  the  importance  of 
accurate  weight  factors  to  accurate  model  outcomes.  Chapter  V  will  present  the 
conclusions  of  the  research,  as  well  as  present  opportunities  for  future  research  related  to 
this  effort. 
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V.  Conclusions 


The  purpose  of  this  chapter  is  to  summarize  the  results  and  impact  of  this 
research,  including  both  the  Value  Focused  Thinking-based  opportunity  assessment 
model  that  was  developed  and  the  comparative  analysis  conducted  using  the  model  to 
evaluate  a  select  group  of  alternatives.  The  initial  research  questions  and  objectives  are 
reviewed,  and  the  results  are  compared  to  those  objectives.  A  discussion  of  both  the 
significance  and  limitations  of  the  research  is  provided,  as  well  as  a  list  of  future 
opportunities  for  research. 

Review  of  Results 

As  presented  in  Chapter  I,  the  objective  of  this  research  project  was  to  develop  a 
decision  support  model  for  the  Civil  Engineer  Commodity  Council  (CECC)  to  assist  with 
opportunity  assessments  of  strategic  sourcing  alternatives.  As  part  of  this  effort,  the 
following  five  research  questions  were  developed: 

1.  What  are  the  objectives  that  Air  Force  leadership  believes  strategic  sourcing 
should  accomplish? 

2.  What  are  the  relative  priorities  of  those  objectives? 

3.  What  variables  predict  potential  efficiencies  in  a  service  or  commodity  contract 
areas? 

4.  Can  the  variables  mentioned  above  be  accurately  measured  with  existing  data  sets 
and  current  data  collection  efforts? 

5.  What  is  the  model  that  accurately  balances  all  objectives  according  to  leadership 
priorities  that  predicts  progress  toward  strategic  sourcing  goals? 
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The  Value  Focused  Thinking  (VFT)  process  selected  as  the  methodology  for  this 
research  addressed  each  of  these  questions  through  the  application  of  the  eight-step 
decision-making  process.  Question  one  was  addressed  through  the  creation  of  the  value 
hierarchy  created  in  step  two  of  the  VFT  process.  Priorities  for  each  of  the  objectives 
were  developed  in  step  six  with  the  assigning  of  weight  factors  for  each  of  the  priorities. 
The  variables  mentioned  in  question  three  took  the  form  of  evaluation  measures  that  were 
used  to  determine  the  suitability  of  each  opportunity  for  furthering  the  objectives  of  the 
strategic  sourcing  program.  While  question  four  was  not  explicitly  linked  to  a  specific 
step  of  the  VFT  process,  the  data  sources  explained  in  Chapter  III  were  identified  to 
evaluate  alternatives  based  on  the  objectives  established  in  the  value  hierarchy.  Finally, 
the  value  equation  developed  using  the  VFT  process  satisfied  the  need  for  a  new  decision 
model  that  considered  the  full  range  of  objectives  of  the  strategic  sourcing  program. 

In  addition  to  the  fact  that  all  the  research  questions  were  addressed,  the  overall 
goal  of  developing  a  decision  model  for  the  CECC  to  assist  with  opportunity  assessments 
was  achieved.  Instead  of  using  an  alternative-focused  system  of  opportunity  assessment 
like  the  one  currently  in  use  by  the  CECC,  an  objective  hierarchy  was  developed  to 
determine  the  broad  range  of  values  and  objectives  important  to  the  Air  Force  and  the 
civil  engineer  community.  By  opening  the  aperture  of  how  opportunities  are  examined 
for  strategic  sourcing  potential  to  encompass  all  aspects  of  the  program  important  to 
senior  leadership,  it  is  possible  to  make  better,  more  informed  decisions  about  the  most 
attractive  opportunities  to  invest  the  time  and  resources  pursuing  for  strategic  sourcing. 

The  value  function  developed  in  the  research  serves  as  a  systematic  means  of 
objectively  analyzing  alternatives  for  the  CECC  opportunity  assessment  process.  The 
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Single  Dimensional  Value  Functions  (SDVF)  for  each  of  the  objectives  effectively 
converts  the  raw  data  collected  on  each  alternative  into  units  of  value  according  to  the 
unique  relationship  between  the  data  and  the  desirability  of  the  alternative.  Using  this 
method  serves  to  limit  subjectivity  in  the  decision-making  process  by  applying  a 
consistent  means  of  ascribing  value  to  each  alternative  based  on  quantitative  or 
qualitative  data  collected  in  a  consistent,  objective  manner. 

In  an  effort  to  validate  the  model  developed  in  the  research,  nine  alternatives  were 
evaluated.  The  alternatives  consisted  of  a  mixture  of  alternatives  previously  evaluated  in 
the  CECC  Commodity  Management  Plan  (CMP),  alternatives  currently  under  review  that 
were  not  included  in  the  CMP,  and  one  alternative  not  actively  under  consideration  by  the 
CECC.  Data  were  collected  for  each  of  the  alternatives,  and  value  scores  were  assigned 
according  to  the  value  equation  developed  in  the  research.  The  resulting  value  scores 
provided  a  much  different  result  compared  to  the  original  CMP  rankings.  The  addition  of 
the  new  evaluation  criteria  provided  fundamentally  different  information  from  that  of  the 
model  used  to  evaluate  the  alternatives  in  the  CMP.  This  indicates  that  many  factors  not 
considered  in  the  original  CMP  decision  model  have  a  significant  impact  on  the  resulting 
priorities  assigned  to  the  alternatives. 

Due  to  the  fact  that  the  additional  information  that  was  collected  was  directly 
related  to  objectives  of  the  strategic  sourcing  program  and  not  considered  in  the  original 
CMP  analysis,  and  that  the  consideration  of  this  information  resulted  in  fundamentally 
different  results,  it  is  clear  that  further  analysis  of  strategic  sourcing  alternatives  must 
take  into  account  the  full  range  of  strategic  sourcing  objectives  to  make  decisions 
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consistent  with  the  goals  of  the  program.  Incorporation  of  these  additional  objectives  into 
the  decision  method  used  by  the  CECC  is  the  main  recommendation  of  this  research. 

In  addition  to  incorporating  the  full  range  of  program  objectives  into  the  decision¬ 
making  process,  it  is  recommended  that  the  CECC  should  adopt  a  more  systematic  and 
objective  approach  to  opportunity  assessments.  The  imminent  nature  of  the  current 
budget  issues  facing  the  federal  government  have  greatly  increased  the  pressure  for  the 
Enterprise  Sourcing  Group  (ESG)  to  focus  narrowly  on  predicting  reductions  in  direct 
expenses  in  an  effort  to  drive  budget  cuts  for  various  items  across  the  Future  Years 
Defense  Program.  This  pressure  has  resulted  in  an  overemphasis  on  first  costs  as  a 
discriminator  in  strategic  sourcing  decisions  to  the  exclusion  of  additional  efficiency 
factors.  In  addition  to  this,  the  pressure  to  produce  results  in  the  form  of  accurate  future 
savings  projections  drives  the  CECC  and  the  ESG  to  spend  an  inordinate  amount  of  time 
and  energy  analyzing  the  few  alternatives  they  have  been  able  to  consider  to  date.  This 
process  greatly  delays  the  execution  of  strategic  sourcing  contracts  to  the  point  of 
negatively  impacting  the  Civil  Engineer  community’s  perception  of  the  effectiveness  of 
the  strategic  sourcing  process. 

Creating  and  implementing  a  defensible  methodology  for  opportunity  assessments 
based  on  scientific  methods  like  VFT  can  help  alleviate  this  issue  by  institutionalizing  the 
full  range  of  strategic  sourcing  objectives  into  an  approved,  standardized  process.  This 
process  can  then  be  executed  without  an  overemphasis  on  first  costs  as  the  “low  hanging 
fruit”  of  the  strategic  sourcing  process.  Educating  and  achieving  the  approval  of  the 
model  by  senior  leadership  is  critical  to  this  concept. 
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Limitations 


While  the  decision  model  developed  in  the  research  is  useful  for  evaluating 
strategic  sourcing  alternatives,  there  are  limitations  to  its  effectiveness.  The  quality  of 
the  data  used  to  evaluate  the  alternatives  is  of  utmost  importance  to  the  quality  of  the 
results  produced  by  the  model.  Threats  to  the  validity  of  the  data  used  for  analysis  exist 
for  both  the  qualitative  and  quantitative  data  analyzed  by  the  model.  In  particular,  the 
quantitative  data  used  from  the  Commander’s  Resource  Information  System  (CRIS)  to 
calculate  first  costs  and  operations  and  maintenance  costs  seemed  to  be  plagued  with 
errors.  While  the  monetary  amounts  listed  for  each  item  in  the  database  were  very 
accurate,  the  supporting  data  fields  that  characterized  the  nature  of  the  expenses  were  not. 
It  seemed  that  the  consistency  and  accuracy  of  the  data  describing  the  type  of  expenses 
varied  as  much  as  the  users  who  generated  the  data.  This  inevitably  has  caused  the  cost 
data  used  in  the  model  to  be  inaccurate.  Utilizing  the  same  data  for  alternatives  that  were 
used  in  the  CMP  analysis  was  an  attempt  to  mitigate  the  effect  of  this  bias  on  the 
comparison  of  the  model  results.  The  fact  that  the  original  CMP  decision  model  relied 
much  more  completely  on  this  inaccurate  data  than  the  model  developed  in  this  research 
adds  more  credence  to  the  need  to  implement  additional  factors  for  consideration  into  the 
decision  model. 

In  addition  to  limitations  due  to  the  quantitative  data  used  in  the  decision  model, 
because  the  qualitative  data  used  in  the  model  relied  on  personal  opinions  of  a  few 
subject  matter  experts,  the  personal  biases  of  the  experts  consulted  impact  the  results  of 
the  model.  The  most  prominent  instance  of  bias  encountered  during  this  research  was  a 
hostility  bias  against  the  strategic  sourcing  concept  itself.  Due  to  the  experts’  personal 
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experiences  with  the  strategic  sourcing  program  over  the  past  several  years,  opinions  to 
the  effectiveness  of  both  the  strategic  sourcing  concept  and  its  implantation  methods  in 
the  Air  Force  have  developed.  For  some  of  the  experts,  these  opinions  may  have 
influenced  the  answers  to  questions  posed  during  the  interview  process  to  limit  any 
perceived  credit  ascribed  to  strategic  sourcing.  To  limit  the  effect  of  this  bias,  the 
interview  questions  were  designed  to  compel  the  experts  to  quantify  their  opinions  in  an 
objective  manner.  Questions  were  standardized  between  the  different  interviews,  and 
answers  were  limited  to  specific  quantifiable  factors  where  possible.  While  this  served  to 
mitigate  the  effect  of  bias  on  the  part  of  the  experts  consulted,  some  level  of  bias  is 
inevitable  whenever  personal  opinions  are  used  for  data. 

Future  Research  Opportunities 

Through  the  process  of  conducting  this  research,  various  opportunities  for  future 
research  related  to  the  VFT  opportunity  assessment  model  and  strategic  sourcing  in 
general  were  identified.  One  major  complication  with  completing  an  accurate  analysis  of 
services  and  commodities  in  use  in  federal  government  acquisitions  is  the  poor  quality  of 
spend  data  available.  Exploring  new  ways  to  capture  opportunity  data  related  to  cost  that 
are  independent  of  the  Commander’s  Resource  Information  System  would  result  in  a 
more  accurate  and  consistent  opportunity  assessment  model. 

While  this  research  conducted  an  analysis  of  several  alternatives  in  the  Civil 
Engineer  community,  conducting  a  systematic  analysis  of  all  commodity  and  service 
contract  areas  would  identify  new  opportunities  for  strategic  sourcing  that  have  not  yet 
been  considered.  In  addition,  the  methodology  used  in  this  research  can  be  used  to 
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develop  similar  models  for  each  of  the  other  Commodity  Councils.  Finally,  further 
research  in  improving  the  VFT  opportunity  assessment  model  developed  in  this  research 
can  be  further  refined  to  more  effectively  evaluate  strategic  sourcing  alternatives  for  the 
CECC. 
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Appendix  A.  ESG  First  Iteration  Value  Hierarchy 
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Appendix  B.  Subject  Matter  Expert  Questionnaire  Template 


Strategic  Sourcing  Opportunity  Assessment  Research  Questionnaire 


Purpose:  Current  methods  in  use  to  assess  strategic  sourcing  opportunities  focus 
primarily  on  accounting  records  and  fail  to  account  for  the  full  spectrum  of  strategic 
sourcing  objectives.  This  research  effort  will  result  in  a  decision  support  tool  that  will 
better  assist  engineers  in  comprehensively  evaluating  strategic  sourcing  opportunities. 
This  questionnaire  is  part  of  a  research  study  that  will  ask  you  about  your  professional 
opinions  related  to  specific  commodity  or  service  areas  with  strategic  sourcing  potential. 
All  answers  to  these  questions  will  be  recorded  in  a  manner  as  to  not  directly  associate 
them  with  your  name.  In  the  final  report,  any  data  gathered  will  be  attributed  to  “Experts 
in  the  Career  Field.” 

Participation:  Your  participation  in  this  data  collection  is  greatly  appreciated  and 
desired.  Though  your  participation  will  be  extremely  helpful  to  this  research,  please 
remember  that  it  is  COMPLETELY  VOLUNTARY. 

Whether  you  decide  to  participate  or  withdraw  from  the  interview  will  have  no 
impact  upon  your  relationship  with  your  unit,  the  United  States  Air  Force,  or  the 
Department  of  Defense. 

Confidentiality:  Remember  that  ALL  ANSWERS  ARE  ANONYMOUS  and  that  no 
one  other  than  the  researchers  will  see  the  data  provided. 

Instructions: 

•  Base  all  of  your  responses  on  your  own  professional  experiences, 
thoughts,  and  knowledge 

•  There  is  no  “right”  answer.  Be  sure  to  state  your  professional  opinion 


Contact  Info:  If  you  have  any  questions,  comments,  or  concerns  about  this  survey, 
please  contact  Capt.  Andrew  Myers  using  the  information  below. 

AFIT/ENV  BLDG  640/Room  104A 
2950  Hobson  Way 

Wright  Patterson  AFB  OH  45433-7765 
Email:  andrew.myers@afit.edu 
Advisor:  alfred.thal@afit.edu 

Phone:  DSN  85-3636  x7401,  commercial  (937)  255-3636  x7401 
Fax:  DSN  986-4699,  commercial  (937)  656-4699 
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Additional  Background: 

The  model  being  used  in  this  project  currently  consists  of  eleven  evaluation 
measures  related  to  the  established  objectives  of  the  strategic  sourcing  program.  For  your 
information,  the  objectives  are  metrics  are  provided  in  the  following  table: 


Objective  Name 

Evaluation  Measure 

Create  Rate  Savings 

Reduce  first  costs 

Current  annual  spend  for  this  alternative 

Reduce  O&M  costs 

Current  annual  O&M  spend  for  this  alternative 

Reduce  final/replacement  costs 

SME  evaluation  for  a  potential  final  cost  savings 

Improve  quality  of  commodities 
and  services 

SME  evaluation  of  current  quality  problems 

Improve  processes 

Reduce  touch  time 

Total  number  of  annual  contracts  for  this  alternative  by 
type 

Leverage  new  technology 

SME  evaluation  of  new  technology  potential 

Streamline  regulations 

SME  evaluation  of  current  regulations 

Minimize  small  business  risk 

Total  percentage  of  all  annual  contracts  for  this 
alternative  that  are  given  to  a  small  business 

Manage  Demand 

Reduce  utility  expenses 

SME  evaluation  of  resource  savings  potential 

Establish  standard  solutions 

SME  evaluation  of  number  of  solutions  in  use  for  this 
requirement 

Establish  COLs 

SME  evaluation  of  COL  feasibility  ( service  contract 
areas  only) 

Four  of  these  objectives  lend  themselves  to  direct,  quantitative  measurement  based  on 
existing  data  sources.  The  other  seven  objectives,  while  still  important  aspects  of  a 
comprehensive  evaluation  of  strategic  sourcing  opportunities,  do  not.  This  questionnaire 
attempts  to  capture  an  indirect,  qualitative  evaluation  of  specific  strategic  sourcing 
opportunities  for  these  seven  objectives  based  on  your  expert  opinions.  There  is  also  a 
question  at  the  end  asking  you  to  provide  any  additional  commodities  or  services  that  you 
think  would  benefit  from  strategic  sourcing.  If  you  have  any  questions,  please  don’t  hesitate 
to  shoot  me  an  e-mail  at  andrew.myers@us.af.mil.  Thanks  for  your  time  and  support  for  this 
research  effort! 
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Questions: 


1.  One  element  of  analyzing  the  total  life-cycle  cost  impact  of  strategic  sourcing  is  the 
final  cost.  These  costs  include  demolition,  disposal,  environmental  remediation,  and 
other  related  expenses.  Regarding  the  strategically-sourced  solution  that  you  feel  would 
be  the  most  effective  at  efficiently  meeting  requirements,  final  costs  would  be  reduced  by 
an  amount  that  is: 

Negligible  (0-2%),  Marginal  (3-5%),  Moderate  (5-10%),  Significant  (10-25%),  or 
Dramatic  (26%+) 

2.  An  optimal  strategic  sourcing  solution  should  strive  to  improve  the  quality  of  the 
commodity  or  service  being  procured.  Regarding  your  feelings  on  CURRENT  quality 
issues  present  in  this  commodity/service  area,  quality-related  problem  are  brought  to  your 
attention: 

Never,  Rarely,  Occasionally,  Often,  Constant 

3.  Strategic  sourcing  presents  an  opportunity  to  rapidly  leverage  new  technologies  across 
the  entire  Air  Force  that  will  improve  efficiency.  Do  you  feel  there  is  a  new,  currently 
untapped  technology  related  to  this  commodity/service  area  that  can  be  utilized  to 
improve  efficiency? 

Yes/No 

4.  In  your  opinion,  do  regulations,  published  guidance,  and/or  standards  related  to  this 
commodity/service  area  require  updating  or  consolidating? 

Yes/No 

5.  Regarding  your  opinion  of  the  optimal  strategically- sourced  solution  for  this 
commodity/service  area,  this  solution  would  reduce  utility  usage/costs  by  an  amount  that 
is: 

Negligible  (0-2%),  Marginal  (3-5%),  Moderate  (5-10%),  Significant  (10-25%),  or 
Dramatic  (26%+) 

6.  One  way  strategic  sourcing  can  generate  efficiencies  across  the  Air  Force  is  by 
standardizing  the  commodities/services  that  we  are  using  to  meet  requirements. 

Regarding  your  experience  with  CURRENT  acquisitions  related  to  this 
service/commodity  area,  how  many  different  types  of  solutions  are  currently  used  for  this 
particular  application? 

1,  1-4,  5-10,  11  + 
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7.  Regarding  your  opinion  of  the  optimal  strategically- sourced  solution  for  this 
commodity/service  area,  COLs  could  result  in  efficiency  improvements  that  are: 

Negligible  (0-2%),  Marginal  (3-5%),  Moderate  (5-10%),  Significant  (10-25%), 
or  Dramatic  (26%+) 

8.  Do  you  know  of  any  commodities  or  services  that  would  be  great  candidates  for  a 
strategically-sourced  solution?  If  so,  please  describe  your  idea  below  and  fill  out  an 
additional  questionnaire  evaluating  it. 
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